scrapy/scrapy

Scrapy seems to fail to load some sites when using proxy or user agent middleware? 4 days ago

help wanted

scrapy/scrapy

63.2K

Scrapy, a fast high-level web crawling & scraping framework for Python.

Python

#crawler#crawling#framework#hacktoberfest#python#scraping#web-scraping#web-scraping-python

"Content-Encoding" header gets stripped from response headers 13 days ago

enhancement good first issue

scrapy/scrapy

63.2K

Scrapy, a fast high-level web crawling & scraping framework for Python.

Python

#crawler#crawling#framework#hacktoberfest#python#scraping#web-scraping#web-scraping-python

Improve DNS caching 23 days ago

enhancement help wanted patch available in progress

scrapy/scrapy

63.2K

Scrapy, a fast high-level web crawling & scraping framework for Python.

Python

#crawler#crawling#framework#hacktoberfest#python#scraping#web-scraping#web-scraping-python

LogCounterHandler should only handle messages from self.crawler about 1 month ago

help wanted logging

scrapy/scrapy

63.2K

Scrapy, a fast high-level web crawling & scraping framework for Python.

Python

#crawler#crawling#framework#hacktoberfest#python#scraping#web-scraping#web-scraping-python

Warning messages to add in the CSV exporter when there's any field dropped 2 months ago

enhancement help wanted

scrapy/scrapy

63.2K

Scrapy, a fast high-level web crawling & scraping framework for Python.

Python

#crawler#crawling#framework#hacktoberfest#python#scraping#web-scraping#web-scraping-python

CrawlTestCase.test_start_requests_lazyness is flaky 2 months ago

bug good first issue CI

scrapy/scrapy

63.2K

Scrapy, a fast high-level web crawling & scraping framework for Python.

Python

#crawler#crawling#framework#hacktoberfest#python#scraping#web-scraping#web-scraping-python

Missing the Scrapy entry in Wikipedia in many languages 2 months ago

enhancement good first issue docs

scrapy/scrapy

63.2K

Scrapy, a fast high-level web crawling & scraping framework for Python.

Python

#crawler#crawling#framework#hacktoberfest#python#scraping#web-scraping#web-scraping-python

Exporters should support nested scrapy.Item objects 3 months ago

enhancement help wanted

scrapy/scrapy

63.2K

Scrapy, a fast high-level web crawling & scraping framework for Python.

Python

#crawler#crawling#framework#hacktoberfest#python#scraping#web-scraping#web-scraping-python

Restarting a crawl using with JOBDIR overwrites feed files that use batch_id 3 months ago

help wanted

scrapy/scrapy

63.2K

Scrapy, a fast high-level web crawling & scraping framework for Python.

Python

#crawler#crawling#framework#hacktoberfest#python#scraping#web-scraping#web-scraping-python

Python zipapp .pyz archive - problem with loading scrapy 3 months ago

bug help wanted

scrapy/scrapy

63.2K

Scrapy, a fast high-level web crawling & scraping framework for Python.

Python

#crawler#crawling#framework#hacktoberfest#python#scraping#web-scraping#web-scraping-python

Crawler logs are cut by day 3 months ago

enhancement good first issue docs

scrapy/scrapy

63.2K

Scrapy, a fast high-level web crawling & scraping framework for Python.

Python

#crawler#crawling#framework#hacktoberfest#python#scraping#web-scraping#web-scraping-python

Exception in image/files pipeline are quietly suppressed 3 months ago

bug help wanted media pipelines

scrapy/scrapy

63.2K

Scrapy, a fast high-level web crawling & scraping framework for Python.

Python

#crawler#crawling#framework#hacktoberfest#python#scraping#web-scraping#web-scraping-python

[Docs] Add Windows note for 'python -m scrapy' alternative in tutorial 3 months ago

enhancement good first issue docs

scrapy/scrapy

63.2K

Scrapy, a fast high-level web crawling & scraping framework for Python.

Python

#crawler#crawling#framework#hacktoberfest#python#scraping#web-scraping#web-scraping-python

Cover `scrapy.extensions.memusage.MemoryUsage` with tests 3 months ago

enhancement good first issue CI

scrapy/scrapy

63.2K

Scrapy, a fast high-level web crawling & scraping framework for Python.

Python

#crawler#crawling#framework#hacktoberfest#python#scraping#web-scraping#web-scraping-python

test_utf16 fails on big-endian architectures 4 months ago

bug help wanted CI

scrapy/scrapy

63.2K

Scrapy, a fast high-level web crawling & scraping framework for Python.

Python

#crawler#crawling#framework#hacktoberfest#python#scraping#web-scraping#web-scraping-python

Pipelines documentation limited 4 months ago

enhancement docs help wanted

scrapy/scrapy

63.2K

Scrapy, a fast high-level web crawling & scraping framework for Python.

Python

#crawler#crawling#framework#hacktoberfest#python#scraping#web-scraping#web-scraping-python

Fix flaky `test_download_with_proxy_https_timeout()` 6 months ago

bug help wanted https CI

scrapy/scrapy

63.2K

Scrapy, a fast high-level web crawling & scraping framework for Python.

Python

#crawler#crawling#framework#hacktoberfest#python#scraping#web-scraping#web-scraping-python

Cannot disable middleware in settings when referencing middleware by its object 7 months ago

enhancement good first issue

scrapy/scrapy

63.2K

Scrapy, a fast high-level web crawling & scraping framework for Python.

Python

#crawler#crawling#framework#hacktoberfest#python#scraping#web-scraping#web-scraping-python

“DeprecationWarning: Attempting to mutate a Context after a Connection was created” with pyOpenSSL 25.1.0 7 months ago

bug help wanted

scrapy/scrapy

63.2K

Scrapy, a fast high-level web crawling & scraping framework for Python.

Python

#crawler#crawling#framework#hacktoberfest#python#scraping#web-scraping#web-scraping-python

Document files inside JOBDIR 7 months ago

enhancement good first issue docs

scrapy/scrapy

63.2K

Scrapy, a fast high-level web crawling & scraping framework for Python.

Python

#crawler#crawling#framework#hacktoberfest#python#scraping#web-scraping#web-scraping-python

`response.follow_all()` problem with `cb_kwargs` getting shared by all requests objects. 7 months ago

enhancement good first issue docs

scrapy/scrapy

63.2K

Scrapy, a fast high-level web crawling & scraping framework for Python.

Python

#crawler#crawling#framework#hacktoberfest#python#scraping#web-scraping#web-scraping-python

Update CI for the codecov/test-results-action change 8 months ago

enhancement good first issue CI

scrapy/scrapy

63.2K

Scrapy, a fast high-level web crawling & scraping framework for Python.

Python

#crawler#crawling#framework#hacktoberfest#python#scraping#web-scraping#web-scraping-python

Remove version markers for old versions from docs 8 months ago

good first issue docs cleanup

scrapy/scrapy

63.2K

Scrapy, a fast high-level web crawling & scraping framework for Python.

Python

#crawler#crawling#framework#hacktoberfest#python#scraping#web-scraping#web-scraping-python

File is not downloading when response.status is 201 8 months ago

bug good first issue media pipelines

scrapy/scrapy

63.2K

Scrapy, a fast high-level web crawling & scraping framework for Python.

Python

#crawler#crawling#framework#hacktoberfest#python#scraping#web-scraping#web-scraping-python

Change default SCHEDULER_PRIORITY_QUEUE to scrapy.pqueues.DownloaderAwarePriorityQueue 8 months ago

good first issue cleanup

scrapy/scrapy

63.2K

Scrapy, a fast high-level web crawling & scraping framework for Python.

Python

#crawler#crawling#framework#hacktoberfest#python#scraping#web-scraping#web-scraping-python

Possible redundant duplicate assertion in test_utils_deprecate.py 9 months ago

good first issue CI cleanup

scrapy/scrapy

63.2K

Scrapy, a fast high-level web crawling & scraping framework for Python.

Python

#crawler#crawling#framework#hacktoberfest#python#scraping#web-scraping#web-scraping-python

Deprecate spider attributes that can be replaced by settings, round 2 9 months ago

good first issue cleanup

scrapy/scrapy

63.2K

Scrapy, a fast high-level web crawling & scraping framework for Python.

Python

#crawler#crawling#framework#hacktoberfest#python#scraping#web-scraping#web-scraping-python

The first rule in a robots.txt with BOM will be ignored 10 months ago

bug good first issue

scrapy/scrapy

63.2K

Scrapy, a fast high-level web crawling & scraping framework for Python.

Python

#crawler#crawling#framework#hacktoberfest#python#scraping#web-scraping#web-scraping-python

Non-typing assert in getdictorlist() 10 months ago

AI Summary: The issue reports that the `getdictorlist()` function in Scrapy's settings module uses `assert` statements that can fail based on user input. It suggests replacing these assertions with `ValueError` exceptions to handle invalid input gracefully.

Complexity: 1/5

bug good first issue

scrapy/scrapy

63.2K

Scrapy, a fast high-level web crawling & scraping framework for Python.

Python

#crawler#crawling#framework#hacktoberfest#python#scraping#web-scraping#web-scraping-python

Option to include all tags and attrs in LinkExtractor with specified exclusions 11 months ago

enhancement good first issue link extraction

scrapy/scrapy

63.2K

Scrapy, a fast high-level web crawling & scraping framework for Python.

Python

#crawler#crawling#framework#hacktoberfest#python#scraping#web-scraping#web-scraping-python

Support for socks5 proxy 11 months ago

enhancement help wanted

scrapy/scrapy

63.2K

Scrapy, a fast high-level web crawling & scraping framework for Python.

Python

#crawler#crawling#framework#hacktoberfest#python#scraping#web-scraping#web-scraping-python

Deprecate spider attributes that can be replaced by settings about 1 year ago

AI Summary: The task involves deprecating four spider attributes in the Scrapy framework (download_delay, download_maxsize, download_warnsize, max_concurrent_requests) by adding deprecation warnings that encourage users to switch to the `custom_settings` or `update_settings()` methods. This requires modifying the Scrapy codebase to include logging of these warnings when the deprecated attributes are used.

Complexity: 4/5

good first issue cleanup

scrapy/scrapy

63.2K

Scrapy, a fast high-level web crawling & scraping framework for Python.

Python

#crawler#crawling#framework#hacktoberfest#python#scraping#web-scraping#web-scraping-python

Deprecate CONCURRENT_REQUESTS_PER_IP about 1 year ago

AI Summary: The task is to deprecate the `CONCURRENT_REQUESTS_PER_IP` setting in the Scrapy web scraping framework. This involves removing the setting, updating documentation to reflect its removal, and potentially suggesting alternative settings (like `CONCURRENT_REQUESTS_PER_DOMAIN`) for users who were relying on it. The challenge lies in providing guidance for users currently using the deprecated setting.

Complexity: 4/5

good first issue cleanup

scrapy/scrapy

63.2K

Scrapy, a fast high-level web crawling & scraping framework for Python.

Python

#crawler#crawling#framework#hacktoberfest#python#scraping#web-scraping#web-scraping-python

Open Issues Need Help