feat: allow import/reimport to wait for deduplication to complete

[valentijnscholten]

Summary

While thinking about what a v3 import API could look like, I realized we're missing one piece of the puzzel in the current (re)import.

By default deduplication runs as an async celery task. The results are not awaited, so the (re)import process can finish before deduplication is complete.
Consequences are:

• The mail notification scan_added contains incorrect statistics
• The API response contains incorrect delta statitistics.

With the recent optimizations on both import and deduplication the chances of these incorrect statistics has gone down. However the most confusing case happens when the deduplication is partly finished. In that case the statistics will contain some duplicates, but not all of them. Example report: #13777

Pro also suffers from this when not using the background-import. Also the pattern in this PR could be used for other Pro features like the current asynchronous prioritization.

The solution in this PR is to introduce a new deduplication_execution_mode enum that controls deduplication execution:

async (default): perform deduplication in the background, so not wait for the results.
sync (block_execution==True): perform deduplication in the foreground/in-line with the (re)import.
async_await: perform deduplication in the background and wait for the results.

The async_await is the new option and could be the new default in the future. The block_execution user profile variable is removed in this PR as it is obselete.
Data is migrated. Using block_execution==True could be used to achieve a similar result, but it would slow the import a lot more because deduplication would run in the foreground. It would also make everything synchronous inluding notifications, pushing to JIRA, etc causing further slowdowns.

The new deduplication_execution_mode can be set in the user profile, but also in the API requests.

In most scenario's the await setting will not lead to significantly longer running import, may only 1 or 2 seconds extra.
The exception is on very busy/overloaded instances where the sync deduplication jobs are queueing up.
For this reason the maximum wait time is 1 minute. After 1 minute the (re)import no longer waits and returns statistics as-is.
A response variable deduplication_complete will be False in this case.

Details

• Add a new import/reimport deduplication execution mode, configurable on the user profile via deduplication_execution_mode (async / async_wait / sync) and overridable per request through a deduplication_execution_mode field on the import and reimport endpoints. The request value takes precedence over the profile, otherwise it falls back to async.
• async_wait (new): deduplication/post-processing is still dispatched to the background, but the request waits for deduplication to finish before responding, so scan_added notifications and the returned statistics reflect the deduplicated state. JIRA push, product grading and other non-deduplication tasks remain asynchronous and are not awaited. async (default, return immediately) and sync (run inline) round out the modes.
• The new field is independent of the existing block_execution flag, which is retained as the global "run all async tasks (notifications, JIRA, grading, ...) in the foreground" switch. A data migration seeds deduplication_execution_mode='sync' for users who had block_execution=True, and the resolver falls back to block_execution → sync, so import behavior is unchanged for them.
• post_process_findings_batch now stores its result (ignore_result=False) so async_wait can join on the dispatched batch. The wait is bounded by the new DD_DEDUPLICATION_ASYNC_WAIT_TIMEOUT setting (default 60s) so a missing or stuck worker degrades to responding anyway rather than hanging.
• Add a deduplication_complete boolean to the import/reimport response indicating whether deduplication had finished by the time the response was produced (true for sync and for async_wait when it completed within the timeout; false for async).
• Make scan_added notifications deduplication-aware: once deduplication has completed, refresh the duplicate status of the affected findings from the database and split the new / reactivated / untouched lists into real and duplicate sub-lists, exclude duplicates from the headline count (so an all-duplicate import sends scan_added_empty), and surface the duplicate groups in the mail and webhook templates. In plain async mode the lists are left untouched, preserving historical behavior.
• The import/reimport response statistics are accurate after the await without further change, since they query findings and annotate the duplicate count directly.
• The browser import/reimport views resolve deduplication_execution_mode from the user's profile (the API resolves it in the serializer), so UI imports honor the profile setting instead of always defaulting to async. (No per-import selector is added to the UI form in this change.)
• Surface deduplication_execution_mode in the user profile form and the user detail view, and add 2.60 upgrade notes.

[github-actions]

This pull request has conflicts, please resolve those before we can evaluate the pull request.

[github-actions]

Conflicts have been resolved. A maintainer will review the pull request shortly.

[github-actions]

This pull request has conflicts, please resolve those before we can evaluate the pull request.

[Maffooch]

Review (code + local e2e). Reviewed the diff and exercised it end-to-end against the running stack.

Verification (all green):

• unittests.test_import_execution_mode → 20 passed; unittests.test_importers_performance → 11 passed / 6 skipped (incl. the new test_deduplication_performance_pghistory_async_wait).
• ruff --config ruff.toml → clean on all changed non-migration files.
• Live e2e against a real Celery worker + broker + django-db result backend (not eager): async_wait import → deduplication_complete=true with correct stats; async → false; invalid mode → 400. So the cross-process AsyncResult.get() join genuinely works with the global CELERY_TASK_IGNORE_RESULT=True — confirming the open question in the helper.py NOTE (see inline comment there).
• Server-rendered UI: view_user shows the new row and the profile form dropdown persists the setting.

One blocking item (the migration collision, #1) and four non-blocking suggestions inline below. Nice feature — the duplicate-aware notification split is a clear improvement.

[Maffooch]

Blocking: migration number collision. dev already ships 0269_normalize_blank_finding_components, which also declares dependencies = [('dojo', '0268_release_authorization_to_pro')]. With this PR there are two 0269 migrations on the same parent → Django reports "Conflicting migrations detected; multiple leaf nodes in the migration graph" on migrate / makemigrations --check, and the merge fails (this is the conflicts-detected label).

Fix: renumber both files and re-chain them after the current leaf:

• 0269_usercontactinfo_deduplication_execution_mode → 0270_..., with dependencies = [('dojo', '0269_normalize_blank_finding_components')]
• 0270_seed_deduplication_execution_mode → 0271_..., with dependencies = [('dojo', '0270_usercontactinfo_deduplication_execution_mode')]

The seed migration itself is fine and idempotent.

[Maffooch]

Non-blocking: the timeout is applied per batch, not per import. This loop does result.get(timeout=timeout) once for each dispatched batch, so worst-case total wait is N_batches × timeout. The PR body, 2.60.md, and the model help_text all say "maximum wait time is 1 minute" — a large import (many batches) with a stuck/overloaded worker can block the request for many minutes. It still degrades safely (never a true hang), so non-blocking, but the advertised bound is wrong.

Mitigation: track a single deadline so the whole join is bounded by one timeout:

import time
deadline = time.monotonic() + timeout
for result in results:
    if result is None or not hasattr(result, "get"):
        continue
    remaining = deadline - time.monotonic()
    if remaining <= 0:
        success = False
        break
    try:
        result.get(timeout=remaining, propagate=False)
    except Exception as e:
        logger.warning("async_wait: error/timeout waiting for post-processing: %s", e)
        success = False

[Maffooch]

Non-blocking: this stores a result row for every batch of every import, in all modes. The result is only consumed in async_wait, but flipping the task default to ignore_result=False makes plain async imports (the default) write rows to the django-db result backend, retained for CELERY_RESULT_EXPIRES (24h). On a busy instance that's a steady stream of otherwise-useless celery_taskmeta rows.

Re: the NOTE's open question — I verified live that the join does work with this override against a real worker + django-db backend + global CELERY_TASK_IGNORE_RESULT=True. The remaining concern is just the storage cost for the non-async_wait modes.

Mitigation: leave the task default alone and request the result only when you'll await it, scoping it to the dispatch in post_processing_dispatch_kwargs / the apply_async call for async_wait, e.g. apply_async(..., ignore_result=False). (Whether .get() still works without the override under your broker is worth one quick check; if it does, the per-call override can be dropped entirely.)

[Maffooch]

Non-blocking: an explicit async_wait is silently downgraded to sync when force_sync=True is also passed. A caller supplying both deduplication_execution_mode='async_wait' and force_sync=True ends up in sync with no warning. The main API/UI paths don't combine them (the serializer/views resolve a clean mode without force_sync), so low risk — but it's a surprising precedence for internal/test callers.

Mitigation: only let force_sync promote to sync when no explicit mode was supplied, e.g. apply the force_sync override before defaulting, or skip it when kwargs.get("deduplication_execution_mode") is a valid mode. At minimum, document that force_sync wins over an explicit mode.

[Maffooch]

Non-blocking: no regression guard for the real cross-process join. This suite (and the perf test, which mocks AsyncResult.get) runs under CELERY_TASK_ALWAYS_EAGER=True, which executes the dispatched task inline on the request's connection — so the actual worker→request join with CELERY_TASK_IGNORE_RESULT=True is never exercised by CI. I confirmed it works by hand in this review, but the behavior the helper.py NOTE worries about is exactly what isn't covered.

Mitigation: consider an opt-in integration test (real broker/worker, e.g. under the integration-test harness) that asserts async_wait blocks until dedup completes and returns deduplication_complete=true. The override_settings/eager docstring here is accurate about why eager is used — just call out that the cross-process path is verified elsewhere/manually.