Asynchronous web crawler with NextJS and PostgreSQL persistence. Captures and stores links based on configurable keywords.

1 stars 0 forks 1 watchers TypeScript MIT License
2 Open Issues Need Help Last updated: Sep 14, 2025

Open Issues Need Help

View All on GitHub

AI Summary: This issue involves creating a comprehensive test suite for the V1 Web Scop API, covering unit, integration, and end-to-end tests. It includes specific unit tests for `CrawlerService`, `SearchController`, and `SearchPresenter`, an integration test for `PostgresSearchJobRepository` against a real database, and a full E2E test for the `/search` feature's CRUD lifecycle. The task also requires configuring a dedicated test environment.

Complexity: 4/5
good first issue

Asynchronous web crawler with NextJS and PostgreSQL persistence. Captures and stores links based on configurable keywords.

TypeScript

AI Summary: This issue proposes implementing a cache manager using Redis with a "Cache-Aside" strategy to optimize read operations for specific IDs (GET /search/:id). The strategy involves checking Redis first, fetching from PostgreSQL and caching if not found, and actively invalidating cached data from Redis whenever the corresponding data is updated or deleted to ensure data freshness.

Complexity: 3/5
good first issue

Asynchronous web crawler with NextJS and PostgreSQL persistence. Captures and stores links based on configurable keywords.

TypeScript