feat: the version-aware scan, from the database to the dashboard (PR-01)

CHANGELOG.md

@@ -26,6 +26,40 @@ what has already shipped, and for nothing else. See `RELEASING.md`.
   split, one what an update closes, and the NVD caveat is always printed. Entries without a
   version are scanned by name, and the scan says which. `list` shows version and CPE. The
   verdict is not stored yet (§5.5): `status`, the API and the web UI are the next step.
+- **The verdict is kept, and the API judges too** (docs/DESIGN.md §5.5). In `cve_software`, per
+  product, with the installed version it was given for: after a new `--version` the old verdict
+  is not shown until a scan judges the new one. `patchradar status` and `/api/cves` show it;
+  `POST /api/watchlist/{name}` takes `version` and `cpe` with the CLI's checks, `GET
+  /api/watchlist` adds `entries`, and `POST /api/scan` judges with the CLI's function
+  (`affected.mark`) and returns the split per product in `verdicts`.
+- **A CVE from KEV or MSRC alone is judged through NVD's record of it.** They carry no
+  `cpeMatch`, so for an entry with an installed version such a CVE could only be "not
+  evaluable" — and KEV's are the ones being exploited. For an entry that can be judged, the
+  scan, CLI and API alike, now asks NVD about each by id (`nvd.records_for`, `cveId=`), paced
+  like every request to NVD, and merges the record in, score included. When NVD cannot be
+  asked, the CVE stays not evaluable and the scan says so (the API in `errors`).
+- **`status` and `/api/cves` rank as a scan does.** CISA's exploitation facts and the CVSS v4.0
+  exploit maturity travelled in the scan's records and were not columns, so the next `status`
+  listed by publication date and an exploited CVE from 2023 sat under every new 5.3. They are
+  stored now (`known_exploited`, `kev_ransomware`, `kev_date_added`, `kev_due_date`,
+  `cvss_exploit_maturity`), refreshed by each scan that brings them — CISA lists a CVE long after
+  NVD publishes it — and the listings are ordered by `priority.sql_order`, built from the rules a
+  scan ranks by; a test runs one set of rows through both. An index on that order makes the full
+  listing faster than the date order was (1.0 ms against 1.45 ms on the benchmark's 500 CVEs).
+  `/api/cves` rows carry `priority`, `priority_reason` and the KEV fields; `status` has a Priority
+  column.
+- **`patchradar import --accept "<name>"` writes what a person confirmed.** `import` proposes and a
+  person confirms, and until the watchlist had the columns the confirmation had nowhere to go.
+  Named once per product, a product the survey identified completely — Chrome, Edge — is watched
+  with the registry's version and its vendor source in `vendor` and `product_id`. One it did not
+  identify (a source without its product, a component, a program nothing answers for) is refused
+  with the survey's reason, and the command exits 1; without `--accept` nothing is written.
+- **The web UI shows what is installed and the verdict.** The watchlist shows the version beside
+  each name; the add form takes a version and a CPE, sent with the CLI's checks, and a refused CPE
+  comes back with NVD's form. The table gains Priority and Verdict columns ("not checked" for a
+  product matched by name), and the CVEs that do not affect the installed version are counted and
+  hidden behind one button, as `scan` does without `--all`. A scan's message gives the split per
+  judged product.
 - **docs/DESIGN.md.** The code referred to "the versionradar design, §5" for these columns; the
   text was never committed and could not be found. It is reconstructed from what `affected.py`,
   `inventory.py` and `tests/test_priority_wiring.py` already committed to.

README.md

@@ -65,7 +65,7 @@ patchradar add notepad --version 8.5.0 --cpe "cpe:2.3:a:notepad-plus-plus:notepa
 patchradar scan notepad --days 365         # what affects 8.5.0, and what is unknown
 patchradar scan notepad --days 365 --all   # also the CVEs that do not affect it
 
-# Show latest CVEs in terminal
+# Show the most urgent stored CVEs, ranked as a scan ranks them
 patchradar status
 
 # Remove software
@@ -88,18 +88,24 @@ affected versions:
 
 - **affected** — listed in the table, with "fixed in X" only where NVD states it;
 - **not evaluable** — NVD has not yet said which products the CVE affects (the
-  usual state for weeks after publication), the CVE came from MSRC or KEV alone,
-  or NVD's versions belong to another scheme: listed, because "cannot tell" is
-  not an all-clear;
+  usual state for weeks after publication), NVD could not be asked about a CVE
+  that came from KEV or MSRC alone — those are looked up in NVD by id — or NVD's
+  versions belong to another scheme: listed, because "cannot tell" is not an
+  all-clear;
 - **not affected** — counted, and listed only with `--all`.
 
 One line gives the split, one says which update closes how many of the
 affected ones — the rest stay open because NVD does not state where they were
 fixed, not because the update falls short — and a count from NVD is always
 followed by the reminder that it counts identifiers, not flaws. Entries without
 a version are scanned by name as before, and the scan names them. The verdict
-is not stored yet, so `status`, the API and the web UI still show CVEs by name
-([docs/DESIGN.md](docs/DESIGN.md) §5.5).
+is stored with the version it was given for: `patchradar status` and
+`/api/cves` show it while that version is still the installed one, and the API
+takes the version and CPE too (`POST /api/watchlist/<name>?version=…&cpe=…`).
+The web UI shows it too: the version beside each watched name, a version and a
+CPE in the add form, Priority and Verdict columns, and the CVEs that do not
+affect the installed version counted and one click away
+([docs/DESIGN.md](docs/DESIGN.md) §5.5, §5.6).
 
 ### Asking Debian what it is actually doing
 
@@ -144,8 +150,13 @@ package, because the machine worth surveying is the one without a checkout: run
 reconfiguration of every installed product, and it changes nothing else either.
 
 `patchradar import` then reports, per product, whether a vendor source can answer
-by version. It writes nothing and guesses nothing: a name cannot be turned into a
-vendor's product id by deduction, so it proposes and a person confirms.
+by version. It guesses nothing: a name cannot be turned into a vendor's product id
+by deduction, so it proposes and a person confirms. Without `--accept` it writes
+nothing; `patchradar import <file> --accept "Google Chrome"`, once per product,
+puts a product listed as watchable by version on the watchlist with its version
+and source. One the survey could not identify completely — a source without the
+product it needs, a component, a program nothing answers for — is refused with
+the survey's reason, however it is asked.
 
 The file boundary in the middle is the point. The analysis is Python, tested
 against recorded snapshots, and runs on a machine that never saw the one being

docs/DESIGN.md

@@ -47,13 +47,21 @@ missing version is an absence, and a default would be a claim:
 | column | holds | set by |
 |---|---|---|
 | `cpe` | the CPE 2.3 name of the product, as NVD writes it: `cpe:2.3:a:notepad-plus-plus:notepad\+\+` | `add --cpe` |
-| `vendor` | the CPE vendor field, `notepad-plus-plus` | derived from `cpe` |
-| `product_id` | the CPE product field, escaped as NVD escapes it, `notepad\+\+` | derived from `cpe` |
+| `vendor` | the vendor source that answers by version — `inventory.py`'s *source*: `chrome`, `edge`, `msrc`, `mozilla` | the inventory import, once confirmed (§5.6) |
+| `product_id` | the product that source needs: `chrome`, `Edge` | the inventory import, once confirmed |
 | `installed_version` | the version installed, `8.5.0` | `add --version` |
-| `channel` | the release channel a vendor source answers for (stable, ESR, LTS) | reserved for the inventory import (§5.6) |
+| `channel` | the release channel the source answers for (stable, ESR, LTS) | the inventory import, once confirmed |
 
 An existing database gets them on the next start, as `cves` got its EPSS columns.
 
+Two routes to an answer by version, kept apart: the CPE, through NVD's
+configurations, which the scan judges with today; and a vendor source, which
+`inventory.py` proposes and a person confirms. The CPE's own vendor and product
+(`notepad-plus-plus`, `notepad\+\+`) are read from `cpe` when the scan needs
+them, not stored. Until 2026-10-17 they were stored in `vendor` and `product_id`,
+by a misreading of this table that `inventory.py` contradicted; no release
+carried that.
+
 ### 5.2 Nothing is guessed
 
 The CPE is typed by a person, never derived from the name. Measured on a real
@@ -78,9 +86,14 @@ name is judged against the installed version with NVD's own `configurations`:
   version of this product is not among them.
 - **not evaluable** — anything that cannot be decided: NVD has not yet stated
   which products the CVE affects (no `configurations`, the usual state of a CVE
-  for weeks after publication), the CVE came from a source other than NVD and
-  NVD's record was not part of the scan, or NVD's versions belong to another
-  scheme than the installed one (`2.07` against `16.0.14334`).
+  for weeks after publication), NVD could not be asked about a CVE that came
+  from KEV or MSRC alone, or NVD's versions belong to another scheme than the
+  installed one (`2.07` against `16.0.14334`).
+
+A CVE that reached the scan only through KEV or MSRC, which carry no
+`cpeMatch`, is looked up in NVD by id (`cveId=`) for an entry that can be judged,
+and NVD's record is merged in — configurations and score with it. Paced like
+every request to NVD; an entry matched by name costs no lookup.
 
 "Not evaluable" never becomes "not affected": for a security tool an answer it
 cannot give is not an absolution.
@@ -104,18 +117,35 @@ The total line keeps its meaning: CVEs found by name.
 
 ### 5.5 What is stored
 
-PR-01a judges in memory, during the scan. Storing the verdict — so that
-`status`, the API and the web UI can show it — needs columns in `cves` and is the
-migration `tests/test_priority_wiring.py` waits for, together with the KEV fields.
+The verdict is kept on the link between a CVE and a product, `cve_software`
+(`verdict`, `verdict_reason`, `verdict_fixed_version`, `judged_version`), because
+one CVE stored once can be affected for one watched product and not for another.
+The latest scan decides it. It is read back only while the installed version is
+still `judged_version`: after `add --version` with a new version, the old verdict
+is not shown until a scan judges the new one. `status` and `/api/cves` show it;
+the API records version and CPE (`POST /api/watchlist/{name}?version=&cpe=`) and
+judges with the same function as the CLI (`affected.mark`).
+
+What decides a rank is stored too: `known_exploited`, `kev_ransomware`,
+`kev_date_added`, `kev_due_date` and the provider's `cvss_exploit_maturity`, in
+`cves`, refreshed by every scan that brings them (CISA lists a CVE long after NVD
+publishes it). `status` and `/api/cves` list most urgent first, with the order
+`priority.sql_order` builds from the rules a scan ranks by, and an index on that
+expression keeps a listing of 50 a read of 50 rows.
 
 ### 5.6 Order of work
 
 1. **PR-01a** — the columns, `add --version --cpe`, the verdict for NVD records
    in the command-line scan, `--all`, the README.
-2. **PR-01b** — KEV and MSRC entries judged through NVD's record of the same
-   CVE; the verdict and the KEV fields stored; the API.
-3. **PR-01c** — the web UI; `patchradar import` writing the mappings a person
-   confirmed, `channel` included.
+2. **PR-01b** — in three parts: the verdict stored and the API (01b-1); KEV
+   and MSRC entries judged through NVD's record of the same CVE (01b-2); the KEV
+   fields stored, with the database-backed listings ranked by priority (01b-3).
+3. **PR-01c** — `patchradar import --accept "<name>"` writing the products a
+   person confirmed, only those the survey identified completely (source and
+   product), into `vendor`, `product_id` and `installed_version` (01c-1);
+   the web UI (01c-2). `channel` stays empty until a source the survey
+   identifies needs one: the two certain today, Chrome and Edge, are matched
+   without it.
 
 ## 6. What PatchRadar does not do
 

docs/nis2-compliance.md

@@ -107,7 +107,7 @@ cosa ricevono:
 <!-- hosts:start -->
 | Host | Quando | Cosa riceve |
 | --- | --- | --- |
-| `services.nvd.nist.gov` | `scan`, da riga di comando o dall'interfaccia web | il nome di ogni software della watchlist (o quello passato a `scan`) come parola chiave di ricerca, con l'intervallo di date; la chiave `NVD_API_KEY`, se impostata, in un'intestazione |
+| `services.nvd.nist.gov` | `scan`, da riga di comando o dall'interfaccia web | il nome di ogni software della watchlist (o quello passato a `scan`) come parola chiave di ricerca, con l'intervallo di date; per un software con versione installata e CPE, l'identificativo di ogni CVE trovata solo da CISA KEV o MSRC (`cveId`); la chiave `NVD_API_KEY`, se impostata, in un'intestazione |
 | `api.msrc.microsoft.com` | `scan` | il mese dei bollettini richiesti; nessun nome di software, il filtro avviene sulla macchina |
 | `www.cisa.gov` | `scan` | nulla: scarica il catalogo KEV intero e lo filtra sulla macchina |
 | `security-tracker.debian.org` | `scan` dall'interfaccia web; `patchradar debian` senza `--file` | nulla: scarica il tracker intero e lo filtra sulla macchina |
@@ -145,8 +145,9 @@ Su disco PatchRadar scrive:
 Lo script `inventory.ps1` è il raccoglitore dell'inventario di una macchina Windows: va
 eseguito a mano, possibilmente come amministratore, e scrive un solo file JSON, nella
 cartella corrente o nel percorso indicato con `-OutFile`. Non contiene comandi di rete e non invia il file a nessuno; il file
-resta dove viene scritto. `patchradar import` lo legge e non scrive nulla né contatta
-nessuno. Per verificare le firme dei driver lo script chiede a Windows
+resta dove viene scritto. `patchradar import` lo legge e non contatta nessuno; scrive
+nella watchlist locale solo i prodotti che una persona conferma con `--accept`, e niente
+senza. Per verificare le firme dei driver lo script chiede a Windows
 `Get-AuthenticodeSignature`, e per farlo Windows può contattare i server delle CA
 indicate nei certificati: quei contatti li decide la configurazione di Windows, non
 PatchRadar.

src/patchradar/affected.py

@@ -26,11 +26,11 @@
 ever assigned an identifier. `Summary.caveat` carries that sentence so a caller
 cannot print "no known vulnerabilities" without it.
 
-**Wired into the command-line scan** for a watchlist entry that has both an
-installed version and a CPE (docs/DESIGN.md §5): `judge` gives each CVE found by
-name one of three verdicts — affected, not affected, not evaluable — and
-`closed_by` says what one update closes. Storing the verdict, and the API, are
-the next step (§5.5); `tests/test_priority_wiring.py` holds that boundary.
+**Wired into both scans**, the CLI's and the API's, for a watchlist entry that
+has both an installed version and a CPE (docs/DESIGN.md §5): `judge` gives each
+CVE found by name one of three verdicts — affected, not affected, not evaluable
+— `mark` applies it to a scan's records, and `closed_by` says what one update
+closes. The verdict is stored with the version it was given for (§5.5).
 """
 from __future__ import annotations
 
@@ -451,3 +451,21 @@ def judge(record: dict, *, version: str, vendor: str, cpe_product: str) -> Verdi
                                       f"compared with {version}")
     return Verdict(NOT_AFFECTED, f"NVD states the affected versions ({stated}); "
                                  f"{version} is not among them")
+
+
+def mark(records: list[dict], installed: tuple[str, str, str] | None) -> None:
+    """Give each scan record its verdict for `installed` — (version, vendor,
+    product), or None for an entry matched by name — and take NVD's
+    configurations off it, which nothing after the verdict reads.
+
+    The one place the CLI and the API judge, so the two cannot disagree. The
+    record keeps the version it was judged for, which is what lets a stored
+    verdict be told apart from one about a version no longer installed.
+    """
+    for record in records:
+        if installed is not None:
+            version, vendor, product = installed
+            verdict = judge(record, version=version, vendor=vendor, cpe_product=product)
+            record.update(verdict=verdict.status, verdict_reason=verdict.reason,
+                          verdict_fixed_version=verdict.fixed_version, judged_version=version)
+        record.pop("configurations", None)