Skip to content

api: escape error strings in batch result JSON (bulk-submit loses all results) - #144

Open
emilstahl wants to merge 1 commit into
urlscan:mainfrom
emilstahl:fix/escape-batch-error-json
Open

api: escape error strings in batch result JSON (bulk-submit loses all results)#144
emilstahl wants to merge 1 commit into
urlscan:mainfrom
emilstahl:fix/escape-batch-error-json

Conversation

@emilstahl

Copy link
Copy Markdown

Bug

bulk-submit on a large URL list fails with:

Error: json: error calling MarshalJSON for type json.RawMessage: invalid character 'h' after object key:value pair

No output at all — the results of every URL in the batch are lost, even though all scans were submitted.

Root cause

api/batch.go BatchResultToRaw() interpolated err.Error() directly into JSON:

errRaw := json.RawMessage(fmt.Sprintf(`{"error": "%s"}`, err.Error()))

Go HTTP client errors embed quotes — *url.Error renders as Get "http://host/": .... When a scan fails with one of these (transient network error, client timeout, connection reset), the raw message becomes:

{"error": "Get "http://host/": context deadline exceeded"}

which is invalid JSON (h of http appears right after the "error": "Get " key:value pair — matching the error message above). The invalid json.RawMessage then makes the final json.MarshalIndent of the whole result array in cmd/scan/bulk.go fail, so the entire batch prints nothing. Small batches usually dodge it because they rarely hit a client-side network error; ~100+ URLs almost always trigger one.

Fix

Marshal the error message with json.Marshal instead of interpolating it. Added a regression test with a *url.Error-style message.

Reproduced with binary urlscan-cli 2026.08.18 on a 108-URL bulk submit; subsets of the same list passed, full list failed deterministically; fixed build returns all 108 results.

BatchResultToRaw interpolated err.Error() directly into a JSON object
via fmt.Sprintf. Go HTTP client errors embed quotes (url.Error format:
`Get "http://host": ...`), producing invalid JSON that aborted the
final marshal of the entire bulk-submit result array, losing all
results. Use json.Marshal for the message instead.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant