Skip to content

fix(quark): recover transient upload control errors - #9643

Open
NovumOrbis wants to merge 1 commit into
AlistGo:mainfrom
NovumOrbis:fix/quark-upload-reliability
Open

fix(quark): recover transient upload control errors#9643
NovumOrbis wants to merge 1 commit into
AlistGo:mainfrom
NovumOrbis:fix/quark-upload-reliability

Conversation

@NovumOrbis

@NovumOrbis NovumOrbis commented Sep 6, 2026

Copy link
Copy Markdown

Problem

Quark can return transient control-plane errors during an otherwise valid
multipart upload.

Two failures were observed in a real Synology Hyper Backup -> AList -> Quark
workload:

  • complete_upload_lock_timeout from /file/upload/finish
  • inner error, requestId ... from /file/update/hash

Previously either error aborted the WebDAV PUT and surfaced to Hyper Backup as
a generic destination/permission-style failure.

Fix

Add bounded, stage-aware recovery around known Quark upload control-plane
transients:

  • annotate upload failures with their stage (pre, hash, part, commit,
    finish)
  • add driver-layer retries only for narrowly matched provider control-plane
    transient messages
  • do not add a driver-layer retry to /file/upload/pre, because replaying pre
    can allocate another task/FID
  • rebuild the part reader for every admitted driver-layer retry
  • retry part/commit control-plane authorization only before the OSS request is
    sent
  • do not classify OSS-formatted errors as driver-layer retryable
  • exhaust the bounded /file/upload/finish retry budget before consulting
    directory visibility
  • after the finish retry budget is exhausted, use exact-FID visibility only as
    a last-resort tiebreaker
  • verify only the regular-file FID allocated by /file/upload/pre
  • never fall back to file name + size for upload identity
  • fail closed when the FID is unavailable, listing fails, or visibility cannot
    be established
  • bypass AList's directory cache and user-facing list filters for the raw
    /file/sort verification probe
  • propagate context.Context into verification requests
  • cap raw visibility pagination at 1,000 pages

Safety

The retry matcher is deliberately bounded and prefix-anchored.

Token extensions such as:

  • complete_upload_lock_timeout_but_other
  • complete_upload_lock_timeouts
  • inner errors, requestId ...
  • inner error_x, requestId ...

are rejected.

Formatted OSS errors such as up status: ..., generic transport errors,
permission failures, and same-name conflicts are not retried by this
driver-layer wrapper.

/file/upload/pre is intentionally not given an additional driver-layer retry.
AList's shared base.RestyClient still has a pre-existing transport-level retry
policy below this code. This PR does not change that behavior and therefore
does not claim that provider or OSS requests can never be replayed at the
transport layer.

For finish recovery, visibility success is accepted only after the bounded
finish retry budget is exhausted and only when /file/sort returns the exact
pre-allocated regular-file FID.

Residual provider-semantic uncertainties remain:

  • repository/public evidence does not formally prove that visibility of the
    pre-allocated FID after a finish error is a documented terminal durability
    signal
  • same-task repeatability of /file/update/hash and /file/upload/finish is
    not formally documented here
  • the shared Resty transport retry policy is pre-existing and remains out of
    scope for this PR

Validation

Regression coverage includes:

  • retry-classifier positive and negative cases
  • token-extension rejection
  • hash transient retry
  • missing-FID fail-closed behavior
  • exact-FID finish recovery
  • finish retries occurring before the visibility fallback
  • stale different-FID rejection
  • bounded pagination and page-cap termination
  • in-flight visibility-request cancellation
  • part auth transient -> auth retried -> OSS sent once by the driver layer
  • commit auth transient -> auth retried -> OSS sent once by the driver layer
  • OSS-formatted part/commit errors not retried by this driver layer

Post-audit fork validation based on this exact code, with only a temporary CI
workflow file added, passed:

  • gofmt
  • go vet ./drivers/quark_uc
  • go test ./drivers/quark_uc
  • go test -race ./drivers/quark_uc
  • go test -race -shuffle=on -count=2 ./drivers/quark_uc
  • go test ./internal/op ./server/webdav
  • repository multi-platform build
  • repository release_docker

Real-world validation note

The DS923+ / Hyper Backup evidence belongs to the earlier combined test
candidate:

26bc4f937a213f44437a5db9a8318e9d317fae53

not to the current upstream-facing head:

ff17adb98c55c8787a66e30c893841cdf3e04751

That earlier real workload:

  • recovered a complete_upload_lock_timeout encountered during finish
  • recovered a later inner error, requestId ... during hash
  • completed the full Hyper Backup task
  • passed Hyper Backup integrity verification
  • survived a DSM package stop/start
  • completed a post-restart incremental backup

The final upstream-facing head further hardens finish recovery by exhausting
the finish retry budget before using exact-FID visibility as a last-resort
tiebreaker. That final fallback ordering has not itself been independently
deployed to the NAS.

Related mkdir-visibility fix: #9642. The two PRs arose from the same Hyper
Backup incident but address separate failure mechanisms.

Add bounded, stage-aware retries for known Quark upload control-plane transients.
Defer finish visibility recovery until the retry budget is exhausted, verify only
the exact upload FID through a raw context-aware /file/sort probe, fail closed
when FID is unavailable, cap visibility pagination, and cover replay/cancellation
safety with regression tests.
@NovumOrbis
NovumOrbis force-pushed the fix/quark-upload-reliability branch from 21e2250 to ff17adb Compare September 6, 2026 16:32
@NovumOrbis
NovumOrbis marked this pull request as ready for review September 7, 2026 02:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant