apache/stormcrawler

Send host info on a specific stream 4 days ago

enhancement help wanted good first issue

apache/stormcrawler

982

A scalable, mature and versatile web crawler based on Apache Storm

Java

#apache-storm#crawler#distributed#java#stormcrawler#web-crawler

[Improvement] Replace linear proxy scan in MultiProxyManager with a lookup map 20 days ago

enhancement core help wanted good first issue

apache/stormcrawler

982

A scalable, mature and versatile web crawler based on Apache Storm

Java

#apache-storm#crawler#distributed#java#stormcrawler#web-crawler

support retry-after in FetcherBolt 20 days ago

help wanted good first issue

apache/stormcrawler

982

A scalable, mature and versatile web crawler based on Apache Storm

Java

#apache-storm#crawler#distributed#java#stormcrawler#web-crawler

JSoup parser to handle text/plain 23 days ago

enhancement core help wanted good first issue

apache/stormcrawler

982

A scalable, mature and versatile web crawler based on Apache Storm

Java

#apache-storm#crawler#distributed#java#stormcrawler#web-crawler

Protocol-okhttp: implement IP filter 23 days ago

enhancement core help wanted good first issue

apache/stormcrawler

982

A scalable, mature and versatile web crawler based on Apache Storm

Java

#apache-storm#crawler#distributed#java#stormcrawler#web-crawler

Fix dead links 29 days ago

documentation help wanted good first issue

apache/stormcrawler

982

A scalable, mature and versatile web crawler based on Apache Storm

Java

#apache-storm#crawler#distributed#java#stormcrawler#web-crawler

Delete redirected pages 2 months ago

core good first issue

apache/stormcrawler

982

A scalable, mature and versatile web crawler based on Apache Storm

Java

#apache-storm#crawler#distributed#java#stormcrawler#web-crawler

New XPATH implementation based on JSON file and organised per hostname or domain 2 months ago

enhancement core good first issue

apache/stormcrawler

982

A scalable, mature and versatile web crawler based on Apache Storm

Java

#apache-storm#crawler#distributed#java#stormcrawler#web-crawler

Capture stats about hosts on status stream post fetching 2 months ago

ideas good first issue

apache/stormcrawler

982

A scalable, mature and versatile web crawler based on Apache Storm

Java

#apache-storm#crawler#distributed#java#stormcrawler#web-crawler

Adaptative URL filter to normalize URLs based on canonical tag 2 months ago

good first issue

apache/stormcrawler

982

A scalable, mature and versatile web crawler based on Apache Storm

Java

#apache-storm#crawler#distributed#java#stormcrawler#web-crawler

SubDocumentsFilterTest should not use a sitemap file 2 months ago

bug good first issue

apache/stormcrawler

982

A scalable, mature and versatile web crawler based on Apache Storm

Java

#apache-storm#crawler#distributed#java#stormcrawler#web-crawler

URLPartitionerBolt to partition on arbitrary metadata value 2 months ago

enhancement core good first issue

apache/stormcrawler

982

A scalable, mature and versatile web crawler based on Apache Storm

Java

#apache-storm#crawler#distributed#java#stormcrawler#web-crawler

Use proxy on a per URL basis 3 months ago

enhancement help wanted good first issue

apache/stormcrawler

982

A scalable, mature and versatile web crawler based on Apache Storm

Java

#apache-storm#crawler#distributed#java#stormcrawler#web-crawler

[Improvement] Documentation: Add PowerShell-specific syntax for Maven archetype generation 3 months ago

documentation good first issue

apache/stormcrawler

982

A scalable, mature and versatile web crawler based on Apache Storm

Java

#apache-storm#crawler#distributed#java#stormcrawler#web-crawler

Move to metrics v2 3 months ago

core help wanted good first issue

apache/stormcrawler

982

A scalable, mature and versatile web crawler based on Apache Storm

Java

#apache-storm#crawler#distributed#java#stormcrawler#web-crawler

Java 20 deprecates all constructors of the class java.net.URL 4 months ago

core help wanted good first issue

apache/stormcrawler

982

A scalable, mature and versatile web crawler based on Apache Storm

Java

#apache-storm#crawler#distributed#java#stormcrawler#web-crawler

Configure mockito as an agent 4 months ago

wish good first issue

apache/stormcrawler

982

A scalable, mature and versatile web crawler based on Apache Storm

Java

#apache-storm#crawler#distributed#java#stormcrawler#web-crawler

[Improvement] The Metadata should handle all the case sensitivities 6 months ago

help wanted good first issue

apache/stormcrawler

982

A scalable, mature and versatile web crawler based on Apache Storm

Java

#apache-storm#crawler#distributed#java#stormcrawler#web-crawler

[Improvement] Work towards Maven 4 Compatibility 8 months ago

help wanted good first issue

apache/stormcrawler

982

A scalable, mature and versatile web crawler based on Apache Storm

Java

#apache-storm#crawler#distributed#java#stormcrawler#web-crawler

Make waitAck Cache Configurable in StatusUpdaterBolt 9 months ago

good first issue OpenSearch

apache/stormcrawler

982

A scalable, mature and versatile web crawler based on Apache Storm

Java

#apache-storm#crawler#distributed#java#stormcrawler#web-crawler

MetadataFilter to filter based on multiple key values 10 months ago

enhancement core help wanted good first issue

apache/stormcrawler

982

A scalable, mature and versatile web crawler based on Apache Storm

Java

#apache-storm#crawler#distributed#java#stormcrawler#web-crawler

Decouple Selenium and OkHttp from Core Module 10 months ago

ideas good first issue Selenium java

apache/stormcrawler

982

A scalable, mature and versatile web crawler based on Apache Storm

Java

#apache-storm#crawler#distributed#java#stormcrawler#web-crawler

Improve Documentation: Add a Beginner-Friendly Tutorial for Getting Started with SC 11 months ago

documentation help wanted good first issue

apache/stormcrawler

982

A scalable, mature and versatile web crawler based on Apache Storm

Java

#apache-storm#crawler#distributed#java#stormcrawler#web-crawler

Investigate and fix NOP logger in tests 11 months ago

help wanted good first issue

apache/stormcrawler

982

A scalable, mature and versatile web crawler based on Apache Storm

Java

#apache-storm#crawler#distributed#java#stormcrawler#web-crawler

[Improvement] Convert Tags in GH actions into SHA about 1 year ago

AI Summary: The task is to update the Apache StormCrawler project's GitHub Actions workflows to use SHA hashes instead of tags for external actions, to comply with Apache's new policy. This involves identifying the external actions used, finding their corresponding SHA hashes, and updating the workflow files accordingly. A tool like eclipse-csi/octopin might assist in this process.

Complexity: 3/5

help wanted good first issue

apache/stormcrawler

982

A scalable, mature and versatile web crawler based on Apache Storm

Java

#apache-storm#crawler#distributed#java#stormcrawler#web-crawler

Open Issues Need Help