feat(sourcing): v2 — API routing + Browserbase escalation, false-positive rate 85% → ~3% - #10
Merged
Conversation
- Amazon /s?k= and AliExpress wholesale URLs marked ok_search_url without HTTP fetch - Digikey product URLs routed to DigikeyClient.product_details (falls through if no creds) - Mouser ProductDetail URLs routed to MouserClient.part_number_search (falls through if no key) - LCSC product URLs routed to local jlcparts SQLite (falls through if DB not cached) - classify_url() dispatcher; all routers return None to fall through to HTTP layer
…scalation - Rotate among Chrome 130 macOS/Windows + Firefox 131 Linux User-Agents - Add Accept, Accept-Language, Accept-Encoding, Sec-Fetch-* headers - 403/429/502/503 → Browserbase escalation via bb fetch (capped at 30/BOM) - 404/410/451 → flag immediately as actionable link rot - Timeout (>10s) → status=slow, not fail - 7-day disk cache at ~/.cache/electronics-stack/sourcing/url_health/<sha256>.json - escalation_budget list[int] passed through check_url for cap enforcement
- test_amazon_search_classified_as_placeholder (4 cases) - test_aliexpress_wholesale_classified_as_placeholder (4 cases) - test_digikey_url_routes_to_api (3 cases incl. discontinued flag) - test_lcsc_url_routes_to_db (3 cases) - test_browserbase_escalation_on_503 (mock subprocess, budget cap) - test_real_link_rot_still_flagged (404, 410) - test_cache_hit / test_expired_cache - test_summary_counts (actionable_failures excludes placeholders + API-routed OKs) All 30 passing.
- TooManyRedirects (eBay redirect loops) → anti-scrape escalation, not link rot - ConnectionError with 'Max retries exceeded' / 'Connection reset' → escalation - bb exit 1 with only node DeprecationWarning in stderr → unchecked_anti_scrape (not flagged as dead link — bb just not configured) - bb auth failure → unchecked_anti_scrape (no false positive)
Owner
Author
|
Superseded by #11, which incorporated this sourcing-v2 implementation and added explicit provider/API boundaries, opt-in Browserbase escalation, MCP flag propagation, import-side-effect contracts, stderr protection, documentation, and passing CI. Closing this branch without merging it separately. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Eliminates anti-scrape noise from BOM URL health audits via 3-layer routing:
bb fetch, capped at 30/BOM, cachedamazon.com/s?k=andaliexpress.com/w/wholesale-classified as user-facing search URLs (not health-checked)Output schema additions (additive, backward-compat)
Per finding:
via,escalated,actionableTop-level:
by_via {...},escalations,actionable_failuresBenchmark — 3 real BOMs (223 URLs total)
False-positive rate: ~85% → ~3%. Each remaining failure is genuinely actionable.
Tests
30 tests covering: search-placeholder classification, API routing, Browserbase escalation, cache hits, real link rot still flagged, summary counts.