Simple web crawler

v1 crawlers (crawl.js, src/main.rs) were developed using node v14.9.0 (npm v6.14.11) and rustc 1.52.1.

v2 crawlers (crawl_v2.js, src/main_v2.rs) require Node >= 20.19 and Rust >= 1.85. See the v2 section below, and benchmarks comparing the two.

To run the crawler

• Node.js

node crawl.js

• Rust

cargo run

Set seed link

• Node.js

SEED_LINK=https://medium.com/tag/web-scraping node crawl.js

• Rust

SEED_LINK=https://medium.com/tag/web-scraping cargo run

Set output file path (default: sites.txt) [Make sure to create the output file first]

• Node.js

touch output.txt
SEED_LINK=https://devurls.com OUTPUT_FILE_PATH=./output.txt node crawl.js

• Rust

touch output.txt
SEED_LINK=https://devurls.com OUTPUT_FILE_PATH=./output.txt cargo run

Persist logs

• Node.js

touch output.txt
SEED_LINK=https://www.youtube.com OUTPUT_FILE_PATH=./output.txt node crawl.js > logs.txt

• Rust

touch output.txt
SEED_LINK=https://www.youtube.com OUTPUT_FILE_PATH=./output.txt cargo run > logs.txt

Delete output and/or log file(s)

make clean

v2 crawlers

Modernized single-file rewrites living next to the originals:

• crawl_v2.js — native fetch + AbortSignal.timeout, cheerio HTML parsing, p-limit concurrency pool, normalize-url de-duplication, robots-parser politeness.
• src/main_v2.rs — async reqwest client, scraper HTML parsing, url relative-link resolution, futures bounded concurrency, std::sync::LazyLock / std::thread::available_parallelism instead of lazy_static / num_cpus.

Both add a MAX_PAGES budget (default 100) so crawls terminate, per-request timeouts, asset-extension filtering, and a final crawl report. The output file is created automatically if missing.

Run the v2 crawlers

• Node.js

npm install
node crawl_v2.js

• Rust

cargo run --bin web_crawler_v2

v2 environment variables

Variable	Default	Notes
SEED_LINK	same defaults as v1	starting URL
OUTPUT_FILE_PATH	./sites.txt	created if missing
MAX_PAGES	100	crawl budget
CONCURRENCY	2× CPU parallelism	max in-flight requests
REQUEST_TIMEOUT_MS / REQUEST_TIMEOUT_SECS	10000 / 10	Node / Rust per-request timeout
RESPECT_ROBOTS	true	Node only; set false to skip robots.txt checks
DISABLE_LOGGING	false	set true for quiet mode

Run the v2 tests

Both suites run unit tests plus an end-to-end crawl against a local fixture HTTP server — no external network access needed.

• Node.js — test.js tests crawl_v2.js with the built-in node:test runner (requires npm install first)

npm install
npm test

(equivalent: node --test test.js)

• Rust — tests/test.rs tests src/main_v2.rs

cargo test

To run a single suite or test by name:

node --test --test-name-pattern="end to end" test.js
cargo test crawls_a_local_site_end_to_end

v2 benchmarks

Both v2 crawlers were run concurrently for 5 minutes from the same seed (https://devurls.com) with identical settings (concurrency = 2× CPU parallelism, 10s timeouts), sampling CPU/RSS every 5 seconds. Full report: benchmark_results.txt.

	Node.js v2 (crawl_v2.js)	Rust v2 (web_crawler_v2, release)
Pages crawled	4,019 (785/min)	6,253 (1,222/min)
Request success rate	79.9%	73.7%
Distinct domains	695	997
CPU (avg / peak)	51% / 196%	10% / 53%
Memory RSS peak	601 MB	92 MB

Rust crawled ~1.6× more pages at ~5× less CPU and ~6.5× less memory. Caveats: the two crawlers competed for the same network bandwidth, and the Node crawler honors robots.txt by default (487 pages skipped), which accounts for part of the gap.