do-web-doc-resolver

Resolve web queries and URLs into compact, LLM-ready Markdown.

Live Demo · Documentation · Report Bug

---

Overview

do-web-doc-resolver fetches web pages and executes search queries, stripping boilerplate and formatting the output into token-dense Markdown for LLM prompt context. It executes an execution cascade across free and paid providers, falling back automatically if a provider fails or returns low-density content.

---

What the Cascade Is

The resolution engine queries providers in tiered priority order, returning upon the first result that satisfies quality thresholds.

Query Cascade Order

1. Semantic Cache: Local SQLite vector cache (sqlite-vec).
2. Free Search Tier: Exa MCP, Exa SDK, Tavily, DuckDuckGo.
3. Paid Search Tier: Serper, Mistral.

URL Cascade Order

1. Semantic Cache: Pre-cached document lookup.
2. Free Static Tier: llms.txt discovery.
3. Free Direct & Lite Tier: Direct HTTP fetch, Jina Reader, Firecrawl.
4. Browser Tier: Mistral Browser.

---

How to Install

Python

Requires Python 3.11 or higher.

git clone https://github.com/d-oit/do-web-doc-resolver.git
cd do-web-doc-resolver
pip install -r requirements.txt

Rust CLI (do-wdr)

Requires Rust 1.80+.

cd cli
cargo build --release

Web UI

Requires Node.js 18+.

cd web
npm install --legacy-peer-deps

---

How to Run

Python CLI

python -m scripts.cli "https://docs.python.org/3/"
python -m scripts.cli "python asyncio taskgroup example"

Python Module

from scripts.resolve import resolve

result = resolve("https://docs.python.org/3/")
print(result["content"])

Rust CLI (do-wdr)

./cli/target/release/do-wdr resolve "https://docs.python.org/3/"
./cli/target/release/do-wdr resolve "rust tokio tutorial"

Web UI

cd web
npm run dev
# Open http://localhost:3000

---

Environment Variables Required

No API keys are required for zero-config operation using free providers. Optional API keys enable additional paid provider tiers:

Environment Variable	Provider	Required	Description
EXA_API_KEY	Exa SDK	No	Enables Exa search and extraction
TAVILY_API_KEY	Tavily	No	Enables Tavily web search
SERPER_API_KEY	Serper	No	Enables Google Search via Serper
FIRECRAWL_API_KEY	Firecrawl	No	Enables Firecrawl scraping
MISTRAL_API_KEY	Mistral AI	No	Enables Mistral Search and Mistral Browser

---

How to Run Tests

Python Test Suite

pytest tests/ -v -m "not live"

Rust Test Suite

cd cli && cargo test

Web UI Playwright Tests

cd web && npx playwright test --project=desktop

Quality Gate Script

./scripts/quality_gate.sh