Krwlr - A Crawler Written In Rust

What it does

This is a simple crawler, it starts to crawl at the given seed and only crawls within the seed domain. The downloaded webpages are stored in ./data/ directory.

Requirements

• Rust
• Cargo

Setup

Create a .env file and set the required environments variables. an example is found in the .env-example file.

How to run

cargo run --release