openagent-eval good first issues (Python)

Add a CODEOWNERS file mapping directories to reviewers 22 days ago

documentation good first issue

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Document the intended mypy-strict enforcement target in DEVELOPMENT.md about 2 months ago

documentation good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Add zero-API-key quickstart using `provider: mock` about 2 months ago

documentation good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Reconcile coverage target: CONTRIBUTING.md says 80%+, CI enforces 75% about 2 months ago

documentation good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Add `oaeval doctor` check for missing optional provider dependencies about 2 months ago

enhancement good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Add `provider_request.md` and `metric_request.md` issue templates about 2 months ago

documentation good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Verify CHANGELOG.md follows Keep a Changelog format and document the process about 2 months ago

documentation good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Seed one good-first-issue per top-level module for newcomer discoverability about 2 months ago

good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Document `provider: mock` / `retriever: mock` as a first-class offline-testing path about 2 months ago

documentation good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

docs: Fix Python version inconsistency in CONTEXT.md 2 months ago

documentation good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Add common evaluation workflow examples to docs/examples.md 2 months ago

enhancement good first issue in-progress

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Add unit tests for ReportManager.reconstruct() edge cases 3 months ago

enhancement good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Add CLI exit codes documentation to docs/cli.md 3 months ago

documentation good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Add a docs/config-reference.md page documenting the full YAML schema 3 months ago

enhancement good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Write a plugin development guide with step-by-step example 3 months ago

documentation good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Create end-to-end tutorial with free APIs and local embeddings 3 months ago

documentation enhancement good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Add Professional Thumbnail Banner to README for Social Media Sharing 3 months ago

documentation enhancement good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Add static/openagent-eval.png showing interactive CLI wizard for README 3 months ago

documentation enhancement good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

[DOCS] [FEATURE] Complete Colab Notebook Tutorial + Video Walkthrough 3 months ago

documentation enhancement good first issue help wanted

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Add unit tests for the udit CLI command 3 months ago

enhancement good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

[GOOD FIRST ISSUE] Create a Python SDK so users can evaluate in 3 lines of code 3 months ago

enhancement good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Fix: Corpus auditor loads .jsonl files as single documents instead of one-per-line 3 months ago

bug good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Add unit tests for the diagnose CLI command 3 months ago

enhancement good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Improve error message when required config fields are missing during validation 3 months ago

enhancement good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Add --version flag to all CLI commands 3 months ago

enhancement good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Fix: MetricTimeoutError drops imeout_seconds=0.0 due to falsy check 3 months ago

bug good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Add a simple progress indicator to oaeval audit command 3 months ago

enhancement good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Fix: Non-ASCII characters silently ignored in content gap analysis 3 months ago

bug good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Add CorpusConfig to the main Config model for unified configuration 3 months ago

enhancement good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Fix: Comparison report footer shows wrong winner when experiment score is lower 3 months ago

bug good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Add input validation for dataset path — show helpful error for missing files 3 months ago

enhancement good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Remove unused imports across the codebase 3 months ago

enhancement good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Add validation for retriever settings field to catch typos early 3 months ago

enhancement good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Add type hints to public API functions in the reports module 3 months ago

enhancement good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Fix: ReportConfig.max_examples is ignored — each report generator hardcodes its limit 3 months ago

bug good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Standardize error messages across all LLM providers for consistency 3 months ago

enhancement good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Document all supported environment variables in a single reference page 3 months ago

documentation good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Add API key setup guide for all LLM providers to docs 3 months ago

documentation good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Add docstrings to all CLI command functions 3 months ago

enhancement good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Add integration test for full evaluation pipeline with mock provider 3 months ago

enhancement good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Add unit tests for the synth CLI command 3 months ago

enhancement good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Fix: commands/__init__.py missing imports for 6 command modules 3 months ago

bug good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Fix: --llm-provider mock flag silently creates OpenAI provider instead of mock 3 months ago

bug good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Fix: ctx variable shadowed from CLIContext to click.Context in main.py 3 months ago

bug good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Fix: Progress bar resets to 1/total_items on completion in run.py 3 months ago

bug good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

Fix: Missing f-string prefix causes {config.timeout} to be printed literally in run.py 3 months ago

bug good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

docs: Add reports output formats documentation page 3 months ago

documentation good first issue

OpenAgentHQ/openagent-eval

9

Open-source CLI framework for evaluating RAG systems and AI agents with standardized metrics, experiment tracking, and reproducible reports.

Python

#cli#llm#metrics#rag#rag-evaluation

OpenAgentHQ/openagent-eval

Open Issues Need Help