fix: recover sync loop after transient Sync API timeout (DX-9488)#112
fix: recover sync loop after transient Sync API timeout (DX-9488)#112harshitha-cstk wants to merge 4 commits into
Conversation
🔒 Security Scan Results
⏱️ SLA Breach Summary
✅ BUILD PASSED - All security checks passed |
There was a problem hiding this comment.
Pull request overview
This PR hardens DataSync’s recovery behavior so transient Sync API/network failures don’t permanently wedge the sync loop, focusing on re-arming the sync gate and preventing unhandled rejections during recovery paths.
Changes:
- Re-arms the sync gate on process-level recovery (
unlock(true)), and resets gate flags / replays cached requests on unlock. - Ensures sync-gate release on
check()failures and wraps event-driven recovery calls with.catch()to avoid unhandled rejections. - Fixes
src/api.tsrequest lifecycle to settle exactly once and to retry timeouts “in place” so a successful retry advances checkpoints.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| src/core/process.ts | Uses unlock(true) during unhandled error recovery to re-arm the sync gate. |
| src/core/inet.ts | Wraps post-reconnect poke() with a catch to prevent unhandled rejections. |
| src/core/index.ts | Re-arms gate inside unlock(refire); releases SQ on check() failure; wraps emitter.on('check') invocation. |
| src/api.ts | Adds settle-once guards and routes socket timeouts through in-place retries to avoid dropped successful retries. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
🔒 Security Scan Results
⏱️ SLA Breach Summary
✅ BUILD PASSED - All security checks passed |
unlock() returned check()/fire(), which reject on a sync failure. Callers (process.ts, q.ts) invoke unlock() fire-and-forget, so that rejection could surface as an unhandled rejection and re-trip the process-level lockdown, undoing the recovery this PR adds. Attach a .catch to both the requestCache replay and the check() call so unlock() handles/logs its own errors and is a safe, self-contained gate toggle.
🔒 Security Scan Results
⏱️ SLA Breach Summary
✅ BUILD PASSED - All security checks passed |
🔒 Security Scan Results
⏱️ SLA Breach Summary
✅ BUILD PASSED - All security checks passed |
Problem
DataSync could permanently stop processing webhook events after a single transient network timeout to the Contentstack Sync API. The webhook endpoint kept returning HTTP 200, but no further content was synced until a manual restart. This is the recurring symptom behind DX-3769 (reopened) and DX-7876 — the retry work in
2.4.0-beta.5hardened the network layer but did not close the underlying recovery defect.Root cause
Traced from customer prod logs against the source. A transient
ETIMEDOUTon the/syncrequest escapes as an unhandled rejection → the process-level lockdown fires → after 10s it callsunlock(). But:unlock()was called withoutrefire=true, so it clearedlockdownbut leftflag.WQ=false/flag.SQunreset.check()'s(!SQ && WQ)gate stayed shut → sync never resumed.flag.SQ=truewithout ever hitting the lockdown handler (non-network content-type schema errors,filterItems/plugin errors, checkpoint-save failures) → silent permanent freeze.api.tsdouble-settled. On socket timeout it bothreject()ed and (viadestroy()→'error') launched a detached retry whose result landed on an already-settled promise and was discarded — so even a successful retry dropped the data and never advanced the checkpoint.emitter.on('check'), post-reconnectpoke()), so their rejections could trip the lockdown.Changes
src/core/process.tsunlock(true)instead ofunlock()src/core/index.tsunlock()fully re-arms the gate (resetSQ, setWQ, replay + clearrequestCache);check()catch always releases the gate (SQ=false) on any error;emitter.on('check')wrapped with.catchsrc/core/inet.tspoke()wrapped with.catchsrc/api.tsresolveOnce/rejectOnce); socket timeout routed through the in-place retry so a successful retry resolves the original requestTesting
npm run build-ts— compiles cleanly (TypeScript 4.9.5).test/core/sync-resume.ts,test/core/sync.ts,test/core/inet.ts,test/core/q.ts.TIMEOUTlowered to force anETIMEDOUT, sync now resumes on the next/notifyinstead of freezing.Risk / backward compatibility
Checklist
🤖 Generated with Claude Code