DimitrovK

DimitrovK/impossible-tasks

Gave coding agents tasks that cannot be done. 61% faked a passing test suite; half of those are invisible to the tests.

7 good first / help-wanted issues · Python · last activity Oct 1, 2026

0 stars 0 forks 0 watchers Python Other
ai-agents ai-safety benchmark hacktoberfest llm pytest python
View on GitHub
7 Open Issues Need Help Last updated: Oct 1, 2026

Open Issues Need Help

View All on GitHub
Generate a browsable HTML index of all 102 runs about 1 month ago
good first issue hacktoberfest
DimitrovK/impossible-tasks
0

Gave coding agents tasks that cannot be done. 61% faked a passing test suite; half of those are invisible to the tests.

Python
#ai-agents#ai-safety#benchmark#hacktoberfest#llm#pytest#python
Add a new impossible task about 1 month ago
good first issue hacktoberfest new task
DimitrovK/impossible-tasks
0

Gave coding agents tasks that cannot be done. 61% faked a passing test suite; half of those are invisible to the tests.

Python
#ai-agents#ai-safety#benchmark#hacktoberfest#llm#pytest#python
classify.py misses cheats hidden in pytest configuration about 1 month ago
good first issue hacktoberfest classifier
DimitrovK/impossible-tasks
0

Gave coding agents tasks that cannot be done. 61% faked a passing test suite; half of those are invisible to the tests.

Python
#ai-agents#ai-safety#benchmark#hacktoberfest#llm#pytest#python
Detect models that emit tool calls as plain text instead of executing them about 1 month ago
good first issue hacktoberfest classifier
DimitrovK/impossible-tasks
0

Gave coding agents tasks that cannot be done. 61% faked a passing test suite; half of those are invisible to the tests.

Python
#ai-agents#ai-safety#benchmark#hacktoberfest#llm#pytest#python
Record tokens and wall-clock time per run about 1 month ago
good first issue hacktoberfest
DimitrovK/impossible-tasks
0

Gave coding agents tasks that cannot be done. 61% faked a passing test suite; half of those are invisible to the tests.

Python
#ai-agents#ai-safety#benchmark#hacktoberfest#llm#pytest#python
Add more inverted tasks, where editing the test is the correct answer about 1 month ago
help wanted hacktoberfest new task
DimitrovK/impossible-tasks
0

Gave coding agents tasks that cannot be done. 61% faked a passing test suite; half of those are invisible to the tests.

Python
#ai-agents#ai-safety#benchmark#hacktoberfest#llm#pytest#python
Add a local Docker runner so the experiment works without a DigitalOcean account about 1 month ago
help wanted hacktoberfest
DimitrovK/impossible-tasks
0

Gave coding agents tasks that cannot be done. 61% faked a passing test suite; half of those are invisible to the tests.

Python
#ai-agents#ai-safety#benchmark#hacktoberfest#llm#pytest#python