ai-robots-txt

ai-robots-txt/ai.robots.txt

A list of AI agents and robots to block.

4.0K stars 168 forks 4.0K watchers Python MIT License

ai crawlers crawling privacy

View on GitHub Website

12 Open Issues Need Help Last updated: Jul 9, 2026

Open Issues Need Help

View All on GitHub

Matching a string of "spider" catches too much 7 days ago

bug help wanted

ai-robots-txt/ai.robots.txt

4.0K

A list of AI agents and robots to block.

Python

#ai#crawlers#crawling#privacy

nginx expressions escapes too much and matches case-insensitive 8 days ago

help wanted

ai-robots-txt/ai.robots.txt

4.0K

A list of AI agents and robots to block.

Python

#ai#crawlers#crawling#privacy

Blocking Mozilla Tabstack 12 days ago

help wanted

ai-robots-txt/ai.robots.txt

4.0K

A list of AI agents and robots to block.

Python

#ai#crawlers#crawling#privacy

Blocks GitLab Visual Studio Code extension about 1 month ago

help wanted

ai-robots-txt/ai.robots.txt

4.0K

A list of AI agents and robots to block.

Python

#ai#crawlers#crawling#privacy

robots.txt falling behind robots.json 2 months ago

bug help wanted

ai-robots-txt/ai.robots.txt

4.0K

A list of AI agents and robots to block.

Python

#ai#crawlers#crawling#privacy

Bingbot Quandry 3 months ago

documentation help wanted

ai-robots-txt/ai.robots.txt

4.0K

A list of AI agents and robots to block.

Python

#ai#crawlers#crawling#privacy

new user-agent 5 months ago

enhancement help wanted

ai-robots-txt/ai.robots.txt

4.0K

A list of AI agents and robots to block.

Python

#ai#crawlers#crawling#privacy

False positive on old Internet Explorer / Trident based browsers containing SLCC in user agent string 6 months ago

help wanted

ai-robots-txt/ai.robots.txt

4.0K

A list of AI agents and robots to block.

Python

#ai#crawlers#crawling#privacy

GrokAppAndroid 7 months ago

help wanted

ai-robots-txt/ai.robots.txt

4.0K

A list of AI agents and robots to block.

Python

#ai#crawlers#crawling#privacy

Test dark visitors update and improve code/test_files/robots.json 8 months ago

good first issue help wanted

ai-robots-txt/ai.robots.txt

4.0K

A list of AI agents and robots to block.

Python

#ai#crawlers#crawling#privacy

python requirements 8 months ago

good first issue help wanted

ai-robots-txt/ai.robots.txt

4.0K

A list of AI agents and robots to block.

Python

#ai#crawlers#crawling#privacy

Block IbouBot 9 months ago

help wanted

ai-robots-txt/ai.robots.txt

4.0K

A list of AI agents and robots to block.

Python

#ai#crawlers#crawling#privacy