[tdruez]

Issues

• Closes: nixpkgs-clarity: Design and implement Scancode pipeline for single nixpkg #1938

Changes

• Add the scan_nix_package pipeline, reusing all the deploy_to_develop steps
• Validate the nixpkgs PURL and check that the commit is on the official master or release-* branches
• Resolve the nixpkgs commit from search.devbox.sh when the PURL has no commit qualifier
• Fetch the source with the nixpkgs patches applied, falling back to the upstream source
• Download the package outputs from cache.nixos.org and extract them with nix-store --restore
• Enable the ELF mapping by default
• Scan the undeployed source files, keeping their not-deployed status
• Create the Nix package from the nixpkgs metadata and store the uncovered licenses in its extra_data
• Delete the Nix store paths and the nixpkgs tarballs cache at the end of each run
• Install nix-bin and zstd in the full Docker image

[chinyeungli]

@tdruez It looks very good. I've made some comments and see if you agree.

[chinyeungli]

If there is no "selected_output", all outputs from self.metadata["outputs"].items() will be fetched such as dev, doc, man, info, etc. Those contain source-like content that maps cleanly to from/ and may produces noise in the D2D report, the binary side is supposed to represent the deployed artifacts, not the whole package.

Should we default to a preferred list instead, e.g. out, debug, and fall back to all outputs if none of those exist?

[chinyeungli]

Should we add a guard when there is no "src" attribute?
Two examples — these .nix files don't have a src attribute:

• migu (only srcs) - https://github.com/NixOS/nixpkgs/blob/5202e105ca0c9c6779fdfa6bb27d4410abc41b5b/pkgs/by-name/mi/migu/package.nix#L11
• papertrail (dontUnpack = true) - https://github.com/NixOS/nixpkgs/blob/5202e105ca0c9c6779fdfa6bb27d4410abc41b5b/pkgs/by-name/pa/papertrail/package.nix#L9

Something like

if !(package ? src) then
   throw "PACKAGE_HAS_NO_SOURCE"
else

and then catch the error in fetch_source ()?

Even without source, the pipeline can still fetch the binary from the cache and create the package from the nixpkgs metadata, so it's worth finishing with a warning rather than failing the run.

If we plan to add a guard, perhaps we can set a self.d2d_enable field to bypass all the d2d_steps.

I think we can skip srcs support in v1 and add it in a later release.