Follow-up from apify/crawlee#3533 (aligning JS enqueueLinks with the Python API, #3409). During review, @vdusek found three remaining behavioral differences between the two implementations that weren't in scope for the JS-side PR. The team agreed (1, 2, 3) that these should be evaluated for alignment on the Python side rather than the JS side. Filing as one issue since all three affect the same enqueue_links/URL-filtering pipeline.
1) Glob case sensitivity
JS uses Minimatch with nocase: true — glob matching is case-insensitive. Python's Glob translates to a regex without re.IGNORECASE (_utils/globs.py) — case-sensitive. The same glob pattern matches different URLs in each library.
Suggested direction (per @B4nan): make Python's glob matching case-insensitive to match JS.
2) Regexp anchoring
Python calls pattern.match(url), which anchors at the start of the string. JS calls regexp.test(url), which searches anywhere in the string. A regex like /\/products\// matches https://x.com/products/1 in JS but not in Python (there it would need to start with https?://…).
Suggested direction (per @B4nan, tentative — "curious about others' opinions"): make Python's regex matching unanchored to match JS.
3) limit vs. transform ordering
Python applies limit in the filter iterator, before transform runs, so requests dropped via a skip action still consume the limit. JS applies limit last (after transform), so skipped requests get backfilled by others. With limit: 5 and a transform skipping 2 of them, Python enqueues 3, JS enqueues 5.
Suggested direction (per @B4nan): apply limit last in Python, to match JS.
None of these are must-fix — the working assumption in the JS PR discussion was that JS's behavior is the more intuitive one and Python should move toward it, but that's up for debate here. See the full review comment for exact line references on the JS side.
Follow-up from apify/crawlee#3533 (aligning JS
enqueueLinkswith the Python API, #3409). During review, @vdusek found three remaining behavioral differences between the two implementations that weren't in scope for the JS-side PR. The team agreed (1, 2, 3) that these should be evaluated for alignment on the Python side rather than the JS side. Filing as one issue since all three affect the sameenqueue_links/URL-filtering pipeline.1) Glob case sensitivity
JS uses
Minimatchwithnocase: true— glob matching is case-insensitive. Python'sGlobtranslates to a regex withoutre.IGNORECASE(_utils/globs.py) — case-sensitive. The same glob pattern matches different URLs in each library.Suggested direction (per @B4nan): make Python's glob matching case-insensitive to match JS.
2) Regexp anchoring
Python calls
pattern.match(url), which anchors at the start of the string. JS callsregexp.test(url), which searches anywhere in the string. A regex like/\/products\//matcheshttps://x.com/products/1in JS but not in Python (there it would need to start withhttps?://…).Suggested direction (per @B4nan, tentative — "curious about others' opinions"): make Python's regex matching unanchored to match JS.
3)
limitvs.transformorderingPython applies
limitin the filter iterator, beforetransformruns, so requests dropped via a skip action still consume the limit. JS applieslimitlast (after transform), so skipped requests get backfilled by others. Withlimit: 5and a transform skipping 2 of them, Python enqueues 3, JS enqueues 5.Suggested direction (per @B4nan): apply
limitlast in Python, to match JS.None of these are must-fix — the working assumption in the JS PR discussion was that JS's behavior is the more intuitive one and Python should move toward it, but that's up for debate here. See the full review comment for exact line references on the JS side.