feat(scan): a CVE from KEV or MSRC alone is judged through NVD's record

CHANGELOG.md

@@ -32,6 +32,12 @@ what has already shipped, and for nothing else. See `RELEASING.md`.
   `POST /api/watchlist/{name}` takes `version` and `cpe` with the CLI's checks, `GET
   /api/watchlist` adds `entries`, and `POST /api/scan` judges with the CLI's function
   (`affected.mark`) and returns the split per product in `verdicts`.
+- **A CVE from KEV or MSRC alone is judged through NVD's record of it.** They carry no
+  `cpeMatch`, so for an entry with an installed version such a CVE could only be "not
+  evaluable" — and KEV's are the ones being exploited. For an entry that can be judged, the
+  scan, CLI and API alike, now asks NVD about each by id (`nvd.records_for`, `cveId=`), paced
+  like every request to NVD, and merges the record in, score included. When NVD cannot be
+  asked, the CVE stays not evaluable and the scan says so (the API in `errors`).
 - **docs/DESIGN.md.** The code referred to "the versionradar design, §5" for these columns; the
   text was never committed and could not be found. It is reconstructed from what `affected.py`,
   `inventory.py` and `tests/test_priority_wiring.py` already committed to.

README.md

@@ -88,9 +88,10 @@ affected versions:
 
 - **affected** — listed in the table, with "fixed in X" only where NVD states it;
 - **not evaluable** — NVD has not yet said which products the CVE affects (the
-  usual state for weeks after publication), the CVE came from MSRC or KEV alone,
-  or NVD's versions belong to another scheme: listed, because "cannot tell" is
-  not an all-clear;
+  usual state for weeks after publication), NVD could not be asked about a CVE
+  that came from KEV or MSRC alone — those are looked up in NVD by id — or NVD's
+  versions belong to another scheme: listed, because "cannot tell" is not an
+  all-clear;
 - **not affected** — counted, and listed only with `--all`.
 
 One line gives the split, one says which update closes how many of the

docs/DESIGN.md

@@ -78,9 +78,14 @@ name is judged against the installed version with NVD's own `configurations`:
   version of this product is not among them.
 - **not evaluable** — anything that cannot be decided: NVD has not yet stated
   which products the CVE affects (no `configurations`, the usual state of a CVE
-  for weeks after publication), the CVE came from a source other than NVD and
-  NVD's record was not part of the scan, or NVD's versions belong to another
-  scheme than the installed one (`2.07` against `16.0.14334`).
+  for weeks after publication), NVD could not be asked about a CVE that came
+  from KEV or MSRC alone, or NVD's versions belong to another scheme than the
+  installed one (`2.07` against `16.0.14334`).
+
+A CVE that reached the scan only through KEV or MSRC, which carry no
+`cpeMatch`, is looked up in NVD by id (`cveId=`) for an entry that can be judged,
+and NVD's record is merged in — configurations and score with it. Paced like
+every request to NVD; an entry matched by name costs no lookup.
 
 "Not evaluable" never becomes "not affected": for a security tool an answer it
 cannot give is not an absolution.

docs/nis2-compliance.md

@@ -107,7 +107,7 @@ cosa ricevono:
 <!-- hosts:start -->
 | Host | Quando | Cosa riceve |
 | --- | --- | --- |
-| `services.nvd.nist.gov` | `scan`, da riga di comando o dall'interfaccia web | il nome di ogni software della watchlist (o quello passato a `scan`) come parola chiave di ricerca, con l'intervallo di date; la chiave `NVD_API_KEY`, se impostata, in un'intestazione |
+| `services.nvd.nist.gov` | `scan`, da riga di comando o dall'interfaccia web | il nome di ogni software della watchlist (o quello passato a `scan`) come parola chiave di ricerca, con l'intervallo di date; per un software con versione installata e CPE, l'identificativo di ogni CVE trovata solo da CISA KEV o MSRC (`cveId`); la chiave `NVD_API_KEY`, se impostata, in un'intestazione |
 | `api.msrc.microsoft.com` | `scan` | il mese dei bollettini richiesti; nessun nome di software, il filtro avviene sulla macchina |
 | `www.cisa.gov` | `scan` | nulla: scarica il catalogo KEV intero e lo filtra sulla macchina |
 | `security-tracker.debian.org` | `scan` dall'interfaccia web; `patchradar debian` senza `--file` | nulla: scarica il tracker intero e lo filtra sulla macchina |

src/patchradar/api/main.py

@@ -21,6 +21,7 @@
 from patchradar.collectors.kev import fetch_cves as kev_fetch
 from patchradar.collectors.msrc import fetch_cves as msrc_fetch
 from patchradar.collectors.nvd import fetch_cves as nvd_fetch
+from patchradar.collectors.nvd import with_records_for_the_rest
 from patchradar.collectors.vuldb import api_key as vuldb_api_key
 from patchradar.collectors.vuldb import fetch_cves as vuldb_fetch
 from patchradar.cpe import CpeError, parse_cpe
@@ -346,6 +347,15 @@ async def _scan_one(sw: str, days: int, errors: list[dict], entry: WatchEntry |
     # and was counted twice — so the two scans gave different totals for the
     # same machine, which is what keeping them in step was meant to prevent.
     collected = merge_by_cve(collected)
+    # As in the CLI: what KEV or MSRC alone found is looked up in NVD by id,
+    # for an entry that can be judged.
+    installed = entry.installed() if entry is not None else None
+    if installed is not None:
+        collected, _, failure = await with_records_for_the_rest(collected, sw)
+        if failure is not None:
+            logger.warning("scan of %r: NVD could not be asked by id: %s", sw, failure)
+            errors.append({"software": sw, "source": failure.source,
+                           "reason": failure.reason, "status": failure.status})
     # FIRST is asked once per package about everything the sources returned,
     # before any of it is stored, so the EPSS fields land in the same row as the
     # CVE rather than in a second pass over the database. It cannot change the
@@ -358,7 +368,6 @@ async def _scan_one(sw: str, days: int, errors: list[dict], entry: WatchEntry |
                        "reason": exc.reason, "status": exc.status})
     # Judged by the function the CLI judges with, before anything is stored, so
     # the verdict lands with the link and the two scans give one answer.
-    installed = entry.installed() if entry is not None else None
     mark(collected, installed)
     if installed is not None and verdicts is not None:
         found = [cve["verdict"] for cve in collected]

src/patchradar/cli.py

@@ -23,7 +23,7 @@
 from patchradar.collectors.msrc import MAX_DAYS_BACK as msrc_max_days
 from patchradar.collectors.msrc import fetch_cves as msrc_fetch
 from patchradar.collectors.nvd import api_key as nvd_api_key
-from patchradar.collectors.nvd import fetch_cves
+from patchradar.collectors.nvd import fetch_cves, with_records_for_the_rest
 from patchradar.collectors.vuldb import SOURCE as vuldb_source
 from patchradar.collectors.vuldb import api_key as vuldb_api_key
 from patchradar.collectors.vuldb import fetch_cves as vuldb_fetch
@@ -399,6 +399,14 @@ async def _scan_target(
     # `severity: "UNKNOWN"`. The count therefore also changes meaning, from
     # records returned to CVEs found — which is what the line claims to say.
     all_cves = merge_by_cve(all_cves)
+    # For an entry the scan can judge, the CVEs that came from KEV or MSRC alone
+    # are looked up in NVD by id: without NVD's configurations they could only
+    # be "not evaluable", and KEV's are the ones being exploited.
+    installed = entry.installed() if entry is not None else None
+    unasked, lookup_failure = 0, None
+    if installed is not None:
+        with _status(f"[cyan]Asking NVD about what other sources found for {safe_target}...[/cyan]"):
+            all_cves, unasked, lookup_failure = await with_records_for_the_rest(all_cves, target)
     # FIRST is asked about the CVEs the scan found, in one request, after the
     # merge and before the sort: `priority` ranks on the EPSS fields, so
     # enriching after the sort would order the report on the facts it had a
@@ -417,7 +425,6 @@ async def _scan_target(
     # each CVE found by name is judged against the version with NVD's own
     # configurations, and the ones that do not affect it are counted, not
     # listed, unless --all asks for them.
-    installed = entry.installed() if entry is not None else None
     judged = installed is not None
     mark(all_cves, installed)
     for cve in all_cves:
@@ -436,6 +443,11 @@ async def _scan_target(
             f"⚠️  [yellow]{escape(str(failure))}[/yellow] — "
             f"results for [bold]{safe_target}[/bold] are incomplete"
         )
+    if lookup_failure is not None:
+        console.print(Text(
+            f"⚠️  NVD could not be asked about {unasked} CVE{'s' if unasked != 1 else ''} "
+            f"from other sources ({lookup_failure}) — not evaluable for {target}", style="yellow"
+        ))
     if epss_failure is not None:
         console.print(
             f"⚠️  [yellow]{escape(str(epss_failure))}[/yellow] — "

src/patchradar/collectors/nvd.py

@@ -9,6 +9,7 @@
 
 from patchradar.collectors.errors import BAD_PAYLOAD, CollectorError, from_http_error
 from patchradar.cvss import exploit_maturity
+from patchradar.priority import merge_by_cve
 
 logger = logging.getLogger(__name__)
 
@@ -336,6 +337,31 @@ async def _fetch_window(
                                  detail=f"empty page at {start_index} of {total}")
 
 
+def _pacer(sleep: Callable[[float], Awaitable[None]], delay: float) -> Callable[[], Awaitable[None]]:
+    """What one call awaits before each of its requests to NVD."""
+    sent = 0
+
+    async def pace() -> None:
+        # Inside one call the previous request has only just been answered, so
+        # the whole interval is waited, as before. Before the first request the
+        # clock is read: a call that starts inside the interval left by the last
+        # request this process sent — whichever keyword sent it — waits for
+        # what is left of it, and one that starts later waits for nothing. A
+        # request that failed counts too: a 403 is the limit speaking.
+        global _last_request_at
+        nonlocal sent
+        if sent:
+            await sleep(delay)
+        elif _last_request_at is not None:
+            remaining = delay - (_monotonic() - _last_request_at)
+            if remaining > 0:
+                await sleep(remaining)
+        sent += 1
+        _last_request_at = _monotonic()
+
+    return pace
+
+
 async def fetch_cves(
     keyword: str, days_back: int = 7, *,
     wait: Callable[[float], Awaitable[None]] | None = None,
@@ -359,30 +385,9 @@ async def fetch_cves(
     caller hands over its own: a test that is about the pacing passes one that
     writes down what it was asked to wait, rather than replacing `_pace`.
     """
-    sleep = wait or _pace
     key = api_key()
     headers = {"apiKey": key} if key else {}
-    delay = KEYED_DELAY_SECONDS if key else KEYLESS_DELAY_SECONDS
-
-    sent = 0
-
-    async def pace() -> None:
-        # Inside one call the previous request has only just been answered, so
-        # the whole interval is waited, as before. Before the first request the
-        # clock is read: a call that starts inside the interval left by the last
-        # request this process sent — whichever keyword sent it — waits for
-        # what is left of it, and one that starts later waits for nothing. A
-        # request that failed counts too: a 403 is the limit speaking.
-        global _last_request_at
-        nonlocal sent
-        if sent:
-            await sleep(delay)
-        elif _last_request_at is not None:
-            remaining = delay - (_monotonic() - _last_request_at)
-            if remaining > 0:
-                await sleep(remaining)
-        sent += 1
-        _last_request_at = _monotonic()
+    pace = _pacer(wait or _pace, KEYED_DELAY_SECONDS if key else KEYLESS_DELAY_SECONDS)
 
     results: list[dict] = []
     async with httpx.AsyncClient(timeout=30.0, headers=headers) as client:
@@ -394,3 +399,56 @@ async def pace() -> None:
                 start += timedelta(days=1)
             await _fetch_window(client, keyword, start, end, pace, results)
     return results
+
+
+
+async def records_for(
+    cve_ids: list[str], keyword: str, *,
+    wait: Callable[[float], Awaitable[None]] | None = None,
+) -> list[dict]:
+    """NVD's records of these CVEs, asked for one id at a time (`cveId=`).
+
+    For the CVEs a scan found through a source that carries no `cpeMatch` —
+    CISA KEV, MSRC — so that an installed version can be judged against them
+    (docs/DESIGN.md §5.3). Paced like every other request to NVD, and an id NVD
+    does not know simply gives nothing. A failure raises with the records
+    already fetched as `partial`.
+    """
+    key = api_key()
+    headers = {"apiKey": key} if key else {}
+    pace = _pacer(wait or _pace, KEYED_DELAY_SECONDS if key else KEYLESS_DELAY_SECONDS)
+    results: list[dict] = []
+    async with httpx.AsyncClient(timeout=30.0, headers=headers) as client:
+        for cve_id in cve_ids:
+            await pace()
+            try:
+                response = await client.get(NVD_API, params={"cveId": cve_id})
+                response.raise_for_status()
+            except httpx.HTTPError as exc:
+                raise from_http_error(SOURCE, exc, partial=results) from exc
+            try:
+                data = response.json()
+            except ValueError as exc:
+                raise CollectorError(SOURCE, BAD_PAYLOAD, status=response.status_code, partial=results) from exc
+            vulnerabilities = data.get("vulnerabilities") if isinstance(data, dict) else None
+            if isinstance(vulnerabilities, list):
+                results.extend(_parse_page(vulnerabilities, keyword))
+    return results
+
+
+async def with_records_for_the_rest(records: list[dict], keyword: str) -> tuple[list[dict], int, CollectorError | None]:
+    """`records`, merged with NVD's record of each one no NVD record came with.
+
+    Returns the merged records, how many CVEs NVD could not be asked about, and
+    the failure that stopped it, if any. The ones left without a record stay as
+    they were, and the verdict on them says so.
+    """
+    missing = [str(r["id"]) for r in records if "configurations" not in r and r.get("id")]
+    if not missing:
+        return records, 0, None
+    try:
+        found = await records_for(missing, keyword)
+        failure = None
+    except CollectorError as exc:
+        found, failure = exc.partial, exc
+    return merge_by_cve(records + found), (len(missing) - len(found) if failure else 0), failure