Every Eval Ever is a shared schema and crowdsourced eval database. It defines a standardized metadata format for storing AI evaluation results — from leaderboard scrapes and research papers to local evaluation runs — so that results from different frameworks can be compared, reproduced, and reused.

agent-evaluation ai-evaluation evaluations infra llm-evaluation
35 Open Issues Need Help Last updated: Jul 5, 2026

Open Issues Need Help

View All on GitHub
enhancement help wanted converters

Every Eval Ever is a shared schema and crowdsourced eval database. It defines a standardized metadata format for storing AI evaluation results — from leaderboard scrapes and research papers to local evaluation runs — so that results from different frameworks can be compared, reproduced, and reused.

Python
#agent-evaluation#ai-evaluation#evaluations#infra#llm-evaluation
good first issue help wanted

Every Eval Ever is a shared schema and crowdsourced eval database. It defines a standardized metadata format for storing AI evaluation results — from leaderboard scrapes and research papers to local evaluation runs — so that results from different frameworks can be compared, reproduced, and reused.

Python
#agent-evaluation#ai-evaluation#evaluations#infra#llm-evaluation
good first issue help wanted

Every Eval Ever is a shared schema and crowdsourced eval database. It defines a standardized metadata format for storing AI evaluation results — from leaderboard scrapes and research papers to local evaluation runs — so that results from different frameworks can be compared, reproduced, and reused.

Python
#agent-evaluation#ai-evaluation#evaluations#infra#llm-evaluation
good first issue help wanted

Every Eval Ever is a shared schema and crowdsourced eval database. It defines a standardized metadata format for storing AI evaluation results — from leaderboard scrapes and research papers to local evaluation runs — so that results from different frameworks can be compared, reproduced, and reused.

Python
#agent-evaluation#ai-evaluation#evaluations#infra#llm-evaluation
good first issue help wanted

Every Eval Ever is a shared schema and crowdsourced eval database. It defines a standardized metadata format for storing AI evaluation results — from leaderboard scrapes and research papers to local evaluation runs — so that results from different frameworks can be compared, reproduced, and reused.

Python
#agent-evaluation#ai-evaluation#evaluations#infra#llm-evaluation
good first issue help wanted

Every Eval Ever is a shared schema and crowdsourced eval database. It defines a standardized metadata format for storing AI evaluation results — from leaderboard scrapes and research papers to local evaluation runs — so that results from different frameworks can be compared, reproduced, and reused.

Python
#agent-evaluation#ai-evaluation#evaluations#infra#llm-evaluation
good first issue help wanted

Every Eval Ever is a shared schema and crowdsourced eval database. It defines a standardized metadata format for storing AI evaluation results — from leaderboard scrapes and research papers to local evaluation runs — so that results from different frameworks can be compared, reproduced, and reused.

Python
#agent-evaluation#ai-evaluation#evaluations#infra#llm-evaluation
good first issue help wanted

Every Eval Ever is a shared schema and crowdsourced eval database. It defines a standardized metadata format for storing AI evaluation results — from leaderboard scrapes and research papers to local evaluation runs — so that results from different frameworks can be compared, reproduced, and reused.

Python
#agent-evaluation#ai-evaluation#evaluations#infra#llm-evaluation
good first issue help wanted

Every Eval Ever is a shared schema and crowdsourced eval database. It defines a standardized metadata format for storing AI evaluation results — from leaderboard scrapes and research papers to local evaluation runs — so that results from different frameworks can be compared, reproduced, and reused.

Python
#agent-evaluation#ai-evaluation#evaluations#infra#llm-evaluation
good first issue help wanted

Every Eval Ever is a shared schema and crowdsourced eval database. It defines a standardized metadata format for storing AI evaluation results — from leaderboard scrapes and research papers to local evaluation runs — so that results from different frameworks can be compared, reproduced, and reused.

Python
#agent-evaluation#ai-evaluation#evaluations#infra#llm-evaluation
good first issue help wanted

Every Eval Ever is a shared schema and crowdsourced eval database. It defines a standardized metadata format for storing AI evaluation results — from leaderboard scrapes and research papers to local evaluation runs — so that results from different frameworks can be compared, reproduced, and reused.

Python
#agent-evaluation#ai-evaluation#evaluations#infra#llm-evaluation
good first issue help wanted

Every Eval Ever is a shared schema and crowdsourced eval database. It defines a standardized metadata format for storing AI evaluation results — from leaderboard scrapes and research papers to local evaluation runs — so that results from different frameworks can be compared, reproduced, and reused.

Python
#agent-evaluation#ai-evaluation#evaluations#infra#llm-evaluation
good first issue help wanted

Every Eval Ever is a shared schema and crowdsourced eval database. It defines a standardized metadata format for storing AI evaluation results — from leaderboard scrapes and research papers to local evaluation runs — so that results from different frameworks can be compared, reproduced, and reused.

Python
#agent-evaluation#ai-evaluation#evaluations#infra#llm-evaluation
good first issue help wanted

Every Eval Ever is a shared schema and crowdsourced eval database. It defines a standardized metadata format for storing AI evaluation results — from leaderboard scrapes and research papers to local evaluation runs — so that results from different frameworks can be compared, reproduced, and reused.

Python
#agent-evaluation#ai-evaluation#evaluations#infra#llm-evaluation
good first issue help wanted

Every Eval Ever is a shared schema and crowdsourced eval database. It defines a standardized metadata format for storing AI evaluation results — from leaderboard scrapes and research papers to local evaluation runs — so that results from different frameworks can be compared, reproduced, and reused.

Python
#agent-evaluation#ai-evaluation#evaluations#infra#llm-evaluation
good first issue help wanted

Every Eval Ever is a shared schema and crowdsourced eval database. It defines a standardized metadata format for storing AI evaluation results — from leaderboard scrapes and research papers to local evaluation runs — so that results from different frameworks can be compared, reproduced, and reused.

Python
#agent-evaluation#ai-evaluation#evaluations#infra#llm-evaluation
good first issue help wanted

Every Eval Ever is a shared schema and crowdsourced eval database. It defines a standardized metadata format for storing AI evaluation results — from leaderboard scrapes and research papers to local evaluation runs — so that results from different frameworks can be compared, reproduced, and reused.

Python
#agent-evaluation#ai-evaluation#evaluations#infra#llm-evaluation
good first issue help wanted

Every Eval Ever is a shared schema and crowdsourced eval database. It defines a standardized metadata format for storing AI evaluation results — from leaderboard scrapes and research papers to local evaluation runs — so that results from different frameworks can be compared, reproduced, and reused.

Python
#agent-evaluation#ai-evaluation#evaluations#infra#llm-evaluation
good first issue help wanted

Every Eval Ever is a shared schema and crowdsourced eval database. It defines a standardized metadata format for storing AI evaluation results — from leaderboard scrapes and research papers to local evaluation runs — so that results from different frameworks can be compared, reproduced, and reused.

Python
#agent-evaluation#ai-evaluation#evaluations#infra#llm-evaluation
good first issue help wanted

Every Eval Ever is a shared schema and crowdsourced eval database. It defines a standardized metadata format for storing AI evaluation results — from leaderboard scrapes and research papers to local evaluation runs — so that results from different frameworks can be compared, reproduced, and reused.

Python
#agent-evaluation#ai-evaluation#evaluations#infra#llm-evaluation
good first issue help wanted

Every Eval Ever is a shared schema and crowdsourced eval database. It defines a standardized metadata format for storing AI evaluation results — from leaderboard scrapes and research papers to local evaluation runs — so that results from different frameworks can be compared, reproduced, and reused.

Python
#agent-evaluation#ai-evaluation#evaluations#infra#llm-evaluation
good first issue help wanted

Every Eval Ever is a shared schema and crowdsourced eval database. It defines a standardized metadata format for storing AI evaluation results — from leaderboard scrapes and research papers to local evaluation runs — so that results from different frameworks can be compared, reproduced, and reused.

Python
#agent-evaluation#ai-evaluation#evaluations#infra#llm-evaluation
good first issue help wanted

Every Eval Ever is a shared schema and crowdsourced eval database. It defines a standardized metadata format for storing AI evaluation results — from leaderboard scrapes and research papers to local evaluation runs — so that results from different frameworks can be compared, reproduced, and reused.

Python
#agent-evaluation#ai-evaluation#evaluations#infra#llm-evaluation
documentation good first issue

Every Eval Ever is a shared schema and crowdsourced eval database. It defines a standardized metadata format for storing AI evaluation results — from leaderboard scrapes and research papers to local evaluation runs — so that results from different frameworks can be compared, reproduced, and reused.

Python
#agent-evaluation#ai-evaluation#evaluations#infra#llm-evaluation
help wanted priority:high datastore adapters converters big-ticket

Every Eval Ever is a shared schema and crowdsourced eval database. It defines a standardized metadata format for storing AI evaluation results — from leaderboard scrapes and research papers to local evaluation runs — so that results from different frameworks can be compared, reproduced, and reused.

Python
#agent-evaluation#ai-evaluation#evaluations#infra#llm-evaluation
enhancement help wanted datastore

Every Eval Ever is a shared schema and crowdsourced eval database. It defines a standardized metadata format for storing AI evaluation results — from leaderboard scrapes and research papers to local evaluation runs — so that results from different frameworks can be compared, reproduced, and reused.

Python
#agent-evaluation#ai-evaluation#evaluations#infra#llm-evaluation
enhancement good first issue validator

Every Eval Ever is a shared schema and crowdsourced eval database. It defines a standardized metadata format for storing AI evaluation results — from leaderboard scrapes and research papers to local evaluation runs — so that results from different frameworks can be compared, reproduced, and reused.

Python
#agent-evaluation#ai-evaluation#evaluations#infra#llm-evaluation
good first issue help wanted schema datastore

Every Eval Ever is a shared schema and crowdsourced eval database. It defines a standardized metadata format for storing AI evaluation results — from leaderboard scrapes and research papers to local evaluation runs — so that results from different frameworks can be compared, reproduced, and reused.

Python
#agent-evaluation#ai-evaluation#evaluations#infra#llm-evaluation

Every Eval Ever is a shared schema and crowdsourced eval database. It defines a standardized metadata format for storing AI evaluation results — from leaderboard scrapes and research papers to local evaluation runs — so that results from different frameworks can be compared, reproduced, and reused.

Python
#agent-evaluation#ai-evaluation#evaluations#infra#llm-evaluation
enhancement help wanted priority:high

Every Eval Ever is a shared schema and crowdsourced eval database. It defines a standardized metadata format for storing AI evaluation results — from leaderboard scrapes and research papers to local evaluation runs — so that results from different frameworks can be compared, reproduced, and reused.

Python
#agent-evaluation#ai-evaluation#evaluations#infra#llm-evaluation
enhancement help wanted validator

Every Eval Ever is a shared schema and crowdsourced eval database. It defines a standardized metadata format for storing AI evaluation results — from leaderboard scrapes and research papers to local evaluation runs — so that results from different frameworks can be compared, reproduced, and reused.

Python
#agent-evaluation#ai-evaluation#evaluations#infra#llm-evaluation
help wanted schema

Every Eval Ever is a shared schema and crowdsourced eval database. It defines a standardized metadata format for storing AI evaluation results — from leaderboard scrapes and research papers to local evaluation runs — so that results from different frameworks can be compared, reproduced, and reused.

Python
#agent-evaluation#ai-evaluation#evaluations#infra#llm-evaluation

Every Eval Ever is a shared schema and crowdsourced eval database. It defines a standardized metadata format for storing AI evaluation results — from leaderboard scrapes and research papers to local evaluation runs — so that results from different frameworks can be compared, reproduced, and reused.

Python
#agent-evaluation#ai-evaluation#evaluations#infra#llm-evaluation

Every Eval Ever is a shared schema and crowdsourced eval database. It defines a standardized metadata format for storing AI evaluation results — from leaderboard scrapes and research papers to local evaluation runs — so that results from different frameworks can be compared, reproduced, and reused.

Python
#agent-evaluation#ai-evaluation#evaluations#infra#llm-evaluation
enhancement good first issue priority:high schema

Every Eval Ever is a shared schema and crowdsourced eval database. It defines a standardized metadata format for storing AI evaluation results — from leaderboard scrapes and research papers to local evaluation runs — so that results from different frameworks can be compared, reproduced, and reused.

Python
#agent-evaluation#ai-evaluation#evaluations#infra#llm-evaluation