scrapy/protego

AI Summary: The task is to fix a bug in a Python robots.txt parser that incorrectly allows partial user-agent matches. The bug is in the `applies_to` method, which currently uses substring matching instead of prefix matching. The fix involves changing the matching logic to use `startswith` and adding a fallback mechanism in `_get_matching_rule_set` to handle cases where no rule matches. This ensures correct user-agent identification and prevents unintended access.

AI Summary: The task is to fix a bug in a Python robots.txt parser. The bug causes incorrect allow/disallow decisions when multiple rules match a URL because it stops at the first match instead of prioritizing the most specific rule. The fix involves modifying the `can_fetch` method to iterate through all rules, selecting the longest matching rule to determine whether a URL is allowed or disallowed.