aviralgarg05

aviralgarg05/agentunit

AgentUnit is a pytest-inspired evaluation harness for autonomous agents and retrieval-augmented generation (RAG) workflows. It helps you describe repeatable scenarios, connect them to your agent stack, and score results with both heuristic and LLM-backed metrics.

1 stars 2 forks 1 watchers Python MIT License
framework pypi-package pytest python
View on GitHub Website
29 Open Issues Need Help Last updated: Dec 6, 2025

Open Issues Need Help

View All on GitHub
Add missing type hints to DatasetCase about 3 hours ago

AI Summary: This issue requests adding explicit type hints to all fields within the `DatasetCase` dataclass located in `src/agentunit/datasets/base.py`. The task involves reviewing the file, applying appropriate Python types like `str` or `list[str]`, and ensuring `from __future__ import annotations` is present. Successful completion requires all fields to be hinted, and both `ruff check` and existing tests to pass.

Complexity: 1/5
enhancement help wanted good first issue
aviralgarg05/agentunit
1

AgentUnit is a pytest-inspired evaluation harness for autonomous agents and retrieval-augmented generation (RAG) workflows. It helps you describe repeatable scenarios, connect them to your agent stack, and score results with both heuristic and LLM-backed metrics.

Python
#framework#pypi-package#pytest#python
Support parallel scenario execution about 19 hours ago
enhancement help wanted
aviralgarg05/agentunit
1

AgentUnit is a pytest-inspired evaluation harness for autonomous agents and retrieval-augmented generation (RAG) workflows. It helps you describe repeatable scenarios, connect them to your agent stack, and score results with both heuristic and LLM-backed metrics.

Python
#framework#pypi-package#pytest#python
Add dataset sampling strategies about 19 hours ago
enhancement help wanted
aviralgarg05/agentunit
1

AgentUnit is a pytest-inspired evaluation harness for autonomous agents and retrieval-augmented generation (RAG) workflows. It helps you describe repeatable scenarios, connect them to your agent stack, and score results with both heuristic and LLM-backed metrics.

Python
#framework#pypi-package#pytest#python
Add HTML report exporter with charts about 19 hours ago
enhancement help wanted
aviralgarg05/agentunit
1

AgentUnit is a pytest-inspired evaluation harness for autonomous agents and retrieval-augmented generation (RAG) workflows. It helps you describe repeatable scenarios, connect them to your agent stack, and score results with both heuristic and LLM-backed metrics.

Python
#framework#pypi-package#pytest#python
Create VS Code extension for AgentUnit configs about 19 hours ago
enhancement help wanted
aviralgarg05/agentunit
1

AgentUnit is a pytest-inspired evaluation harness for autonomous agents and retrieval-augmented generation (RAG) workflows. It helps you describe repeatable scenarios, connect them to your agent stack, and score results with both heuristic and LLM-backed metrics.

Python
#framework#pypi-package#pytest#python
Add scenario tagging and filtering about 19 hours ago
enhancement help wanted
aviralgarg05/agentunit
1

AgentUnit is a pytest-inspired evaluation harness for autonomous agents and retrieval-augmented generation (RAG) workflows. It helps you describe repeatable scenarios, connect them to your agent stack, and score results with both heuristic and LLM-backed metrics.

Python
#framework#pypi-package#pytest#python
Add streaming output for live evaluation progress about 19 hours ago
enhancement help wanted
aviralgarg05/agentunit
1

AgentUnit is a pytest-inspired evaluation harness for autonomous agents and retrieval-augmented generation (RAG) workflows. It helps you describe repeatable scenarios, connect them to your agent stack, and score results with both heuristic and LLM-backed metrics.

Python
#framework#pypi-package#pytest#python
Support custom metrics via YAML configuration about 19 hours ago
enhancement help wanted
aviralgarg05/agentunit
1

AgentUnit is a pytest-inspired evaluation harness for autonomous agents and retrieval-augmented generation (RAG) workflows. It helps you describe repeatable scenarios, connect them to your agent stack, and score results with both heuristic and LLM-backed metrics.

Python
#framework#pypi-package#pytest#python
Add baseline comparison with regression detection about 19 hours ago
enhancement help wanted
aviralgarg05/agentunit
1

AgentUnit is a pytest-inspired evaluation harness for autonomous agents and retrieval-augmented generation (RAG) workflows. It helps you describe repeatable scenarios, connect them to your agent stack, and score results with both heuristic and LLM-backed metrics.

Python
#framework#pypi-package#pytest#python
Add --watch mode for continuous evaluation about 19 hours ago
enhancement help wanted
aviralgarg05/agentunit
1

AgentUnit is a pytest-inspired evaluation harness for autonomous agents and retrieval-augmented generation (RAG) workflows. It helps you describe repeatable scenarios, connect them to your agent stack, and score results with both heuristic and LLM-backed metrics.

Python
#framework#pypi-package#pytest#python
Create pytest plugin for scenario discovery about 19 hours ago
enhancement help wanted
aviralgarg05/agentunit
1

AgentUnit is a pytest-inspired evaluation harness for autonomous agents and retrieval-augmented generation (RAG) workflows. It helps you describe repeatable scenarios, connect them to your agent stack, and score results with both heuristic and LLM-backed metrics.

Python
#framework#pypi-package#pytest#python
Add LLM cost tracking and budgeting about 19 hours ago
enhancement help wanted
aviralgarg05/agentunit
1

AgentUnit is a pytest-inspired evaluation harness for autonomous agents and retrieval-augmented generation (RAG) workflows. It helps you describe repeatable scenarios, connect them to your agent stack, and score results with both heuristic and LLM-backed metrics.

Python
#framework#pypi-package#pytest#python
Add LangGraph integration tests about 19 hours ago
help wanted tests
aviralgarg05/agentunit
1

AgentUnit is a pytest-inspired evaluation harness for autonomous agents and retrieval-augmented generation (RAG) workflows. It helps you describe repeatable scenarios, connect them to your agent stack, and score results with both heuristic and LLM-backed metrics.

Python
#framework#pypi-package#pytest#python
Add result caching for unchanged scenarios about 19 hours ago
enhancement help wanted performance
aviralgarg05/agentunit
1

AgentUnit is a pytest-inspired evaluation harness for autonomous agents and retrieval-augmented generation (RAG) workflows. It helps you describe repeatable scenarios, connect them to your agent stack, and score results with both heuristic and LLM-backed metrics.

Python
#framework#pypi-package#pytest#python
Support async adapters about 19 hours ago
enhancement help wanted
aviralgarg05/agentunit
1

AgentUnit is a pytest-inspired evaluation harness for autonomous agents and retrieval-augmented generation (RAG) workflows. It helps you describe repeatable scenarios, connect them to your agent stack, and score results with both heuristic and LLM-backed metrics.

Python
#framework#pypi-package#pytest#python
Add retry decorator with exponential backoff about 19 hours ago
enhancement help wanted
aviralgarg05/agentunit
1

AgentUnit is a pytest-inspired evaluation harness for autonomous agents and retrieval-augmented generation (RAG) workflows. It helps you describe repeatable scenarios, connect them to your agent stack, and score results with both heuristic and LLM-backed metrics.

Python
#framework#pypi-package#pytest#python
Add basic evaluation example script about 19 hours ago
documentation help wanted good first issue
aviralgarg05/agentunit
1

AgentUnit is a pytest-inspired evaluation harness for autonomous agents and retrieval-augmented generation (RAG) workflows. It helps you describe repeatable scenarios, connect them to your agent stack, and score results with both heuristic and LLM-backed metrics.

Python
#framework#pypi-package#pytest#python
Fix typos in docstrings about 19 hours ago
documentation help wanted good first issue
aviralgarg05/agentunit
1

AgentUnit is a pytest-inspired evaluation harness for autonomous agents and retrieval-augmented generation (RAG) workflows. It helps you describe repeatable scenarios, connect them to your agent stack, and score results with both heuristic and LLM-backed metrics.

Python
#framework#pypi-package#pytest#python
Add docstrings to BaseAdapter methods about 19 hours ago
documentation help wanted good first issue
aviralgarg05/agentunit
1

AgentUnit is a pytest-inspired evaluation harness for autonomous agents and retrieval-augmented generation (RAG) workflows. It helps you describe repeatable scenarios, connect them to your agent stack, and score results with both heuristic and LLM-backed metrics.

Python
#framework#pypi-package#pytest#python
Expose __version__ in agentunit package about 19 hours ago
enhancement help wanted good first issue
aviralgarg05/agentunit
1

AgentUnit is a pytest-inspired evaluation harness for autonomous agents and retrieval-augmented generation (RAG) workflows. It helps you describe repeatable scenarios, connect them to your agent stack, and score results with both heuristic and LLM-backed metrics.

Python
#framework#pypi-package#pytest#python
Improve test coverage for metrics registry about 19 hours ago
help wanted good first issue tests
aviralgarg05/agentunit
1

AgentUnit is a pytest-inspired evaluation harness for autonomous agents and retrieval-augmented generation (RAG) workflows. It helps you describe repeatable scenarios, connect them to your agent stack, and score results with both heuristic and LLM-backed metrics.

Python
#framework#pypi-package#pytest#python
Add py.typed marker for type checker support about 19 hours ago
enhancement help wanted good first issue
aviralgarg05/agentunit
1

AgentUnit is a pytest-inspired evaluation harness for autonomous agents and retrieval-augmented generation (RAG) workflows. It helps you describe repeatable scenarios, connect them to your agent stack, and score results with both heuristic and LLM-backed metrics.

Python
#framework#pypi-package#pytest#python
Add test for markdown report emoji encoding about 19 hours ago
help wanted good first issue tests
aviralgarg05/agentunit
1

AgentUnit is a pytest-inspired evaluation harness for autonomous agents and retrieval-augmented generation (RAG) workflows. It helps you describe repeatable scenarios, connect them to your agent stack, and score results with both heuristic and LLM-backed metrics.

Python
#framework#pypi-package#pytest#python
Add smoke test for suite_template FAQAdapter about 19 hours ago
help wanted good first issue tests
aviralgarg05/agentunit
1

AgentUnit is a pytest-inspired evaluation harness for autonomous agents and retrieval-augmented generation (RAG) workflows. It helps you describe repeatable scenarios, connect them to your agent stack, and score results with both heuristic and LLM-backed metrics.

Python
#framework#pypi-package#pytest#python
Clarify local CI commands in README about 19 hours ago
documentation help wanted good first issue
aviralgarg05/agentunit
1

AgentUnit is a pytest-inspired evaluation harness for autonomous agents and retrieval-augmented generation (RAG) workflows. It helps you describe repeatable scenarios, connect them to your agent stack, and score results with both heuristic and LLM-backed metrics.

Python
#framework#pypi-package#pytest#python
Pin datetime usage to timezone-aware helper about 19 hours ago
help wanted good first issue chore
aviralgarg05/agentunit
1

AgentUnit is a pytest-inspired evaluation harness for autonomous agents and retrieval-augmented generation (RAG) workflows. It helps you describe repeatable scenarios, connect them to your agent stack, and score results with both heuristic and LLM-backed metrics.

Python
#framework#pypi-package#pytest#python
Add sample scenario tests for suite_template.py about 19 hours ago
help wanted good first issue tests
aviralgarg05/agentunit
1

AgentUnit is a pytest-inspired evaluation harness for autonomous agents and retrieval-augmented generation (RAG) workflows. It helps you describe repeatable scenarios, connect them to your agent stack, and score results with both heuristic and LLM-backed metrics.

Python
#framework#pypi-package#pytest#python
Add quickstart instructions for running CI checks locally about 19 hours ago
documentation help wanted good first issue
aviralgarg05/agentunit
1

AgentUnit is a pytest-inspired evaluation harness for autonomous agents and retrieval-augmented generation (RAG) workflows. It helps you describe repeatable scenarios, connect them to your agent stack, and score results with both heuristic and LLM-backed metrics.

Python
#framework#pypi-package#pytest#python
Refactor datetime.utcnow usages to timezone-aware datetime about 19 hours ago
help wanted good first issue chore
aviralgarg05/agentunit
1

AgentUnit is a pytest-inspired evaluation harness for autonomous agents and retrieval-augmented generation (RAG) workflows. It helps you describe repeatable scenarios, connect them to your agent stack, and score results with both heuristic and LLM-backed metrics.

Python
#framework#pypi-package#pytest#python