nixpkgs-clarity: Design and implement Scancode pipeline for single nixpkg

A new pipeline designed specifically for Nix packages, that is aware of the conventions, and unwritten community practices in license and origin documentation.

Here is a scenario for a dedicated pipeline for nixpkg:

The ScanCode input should be a nix PURL with type TBD, or some URL?
The source and binary is downloaded and we run a d2d process between that binary and the source to find any discrepancy
we also should find the source repo and ensure it matches the sources
we should run a source code scan for license (scan codebase) and check that licenses match the metadata
we should also match the source code in the package MatchCode to find borrowed code from a 3rd party

See #2155 for reference on approaches to get packages' metadata

we can use fetchcode to download the source and binary once aboutcode-org/fetchcode#203 is approached and merged

Implement Nix Pipeline #1938

Remove the "rust" references and move check_docker_command to utils.py …

PR created: #2209 and it's ready for review

Enhance efficiency for the pipeline #1938

Better error handling and add back the missing "collect_and_create_co…

Add back the missing f prefix #1938

Fix scan_nix_package availability to accept a local Docker daemon #1938

Revert unwanted changes in d2d.py #1938

Remove unnecessary None condition check in find_uncovered_licenses #1938

Drop dead exception handlers from docker_rm_force #1938

Use SCANCODEIO_FETCH_HEADERS for GitHub API authentication #1938

Return True/False from verify_commit_in_nixpkgs and raise on API fail…

Check trusted branches before listing release branches #1938

Batch codebase resource ids with islice to bound bulk_create memory #…

emove redundant output validation in extract_nar_archive() #1938

Restore the mistakenly removed tests #1938

Rewrite test_scanpipe_license_validation_shape_difference_is_not_a_mi…

Correct typo and write tests without lambda #1938

Move Docker helpers from utils.py to a new container.py #1938

Plan for scan_nix_package version 1

Summary

Takes one Nix PURL as input (e.g. pkg:nix/nixpkgs/hello@2.12.3?system=x86_64-linux&commit=<40-hex>).
nix called directly, no Docker.
Fetches the binary from cache.nixos.org (no local build. Add a project warning and skip D2D if the binary doesn't exist in cache).
Fetches upstream source and applies patches via applyPatches.
Scans the source regardless of whether D2D ran, and compares the declared license against the detected licenses.
Stores the license check result in the package's extra_data.

No Docker, no nixpkgs clone, no local build, no new settings, no new model field, no migration.

Steps

Validate the input
check nix available in get_availability() such as shutil.which("nix")
build the nixpkgs import expression using fetchTarball at the pinned commit (No clone)
fetch_nix_binary (e.g. nix-build --option fallback false) - Nix will fetch from cache.nixos.org and fail otherwise. On failure, add a project warning
extract_nix_binary_to_codebase if binary exist
fetch_nix_source using applyPatches expression
Usual scans + d2d steps
create_package_from_nix_metadata (as current SCTK don't have nix package support)
validate_nix_package_license with the result stored in its extra_data

Out of scope for v1

No local binary builds
No local nixpkgs clone, worktrees, file locks, store garbage collection
No Docker / DinD

Questions to settle

How nix gets installed
Add Dockerfile-nix and docker-compose-nix.yml to install nix in an opt-in image variant, and add corresponding Makefile targets.
The default Dockerfile and docker-compose.yml are untouched so users who don't run this pipeline never pull the heavier image.
Settings
WIth v1, there should be no settings needed.

@chinyeungli

@chinyeungli Here is the v1 plan, implemented and tested in #2291.
Please review the code and let me know if this works for you.

Input

One nixpkgs PURL: pkg:nix/nixpkgs/<attribute path>@<version>?system=<system>

system is required, for example x86_64-linux.
commit is optional: when missing, it is looked up on search.devbox.sh from the version and the system.
output is optional, to fetch a single output of the package.
Nested attribute paths are supported, for example haskellPackages.aeson.

The PURL values are validated against a strict set of characters. The commit must be reachable from the official master or release-* branches, so fork commits are rejected. The GitHub API token can be provided with the existing SCANCODEIO_FETCH_HEADERS setting.

Steps

Validate the PURL, then resolve and verify the nixpkgs commit.
Evaluate nixpkgs at that commit with nix-instantiate and fetchTarball, no local clone: version, description, homepage, meta.license (lists combined with AND), and the store path of each output for the target system.
Fetch the source with the nixpkgs patches applied (nix-build with applyPatches) to from/. When the patches cannot be applied, the upstream source is used, with a project warning that includes the error.
Download each output from cache.nixos.org (.narinfo then the NAR, over HTTP) and extract it with nix-store --restore to to/<output>/. There is no local build: the outputs missing from the cache are reported in a project warning.
Run all the deploy_to_develop steps. The ELF mapping is always enabled. It uses the DWARF paths, so it works best for the packages with a debug output. The other ecosystems can be selected like in deploy_to_develop.
Scan the whole source, including the files not deployed in the binary, which keep their not-deployed status.
Create the Nix package from the nixpkgs metadata, and assign all the codebase resources to it.
License check: the license keys detected in from/ but not part of the declared license are stored in the package extra_data, under uncovered_licenses.
Delete the Nix store paths and the nixpkgs tarballs cache, so the disk usage does not grow across scans.

To match the source with MatchCode.io, add the match_to_matchcode addon pipeline to the same project.

Installation

nix-bin and zstd are installed in the full Docker image, which adds 29 MB. Nix runs in single-user mode as the app user, without the build sandbox. The core image is unchanged, and the pipeline is reported as unavailable when the nix commands are missing.

There are no new settings, no model changes, and no migration.

Tested on

hello@2.12.3: patched source, out output, D2D, license check.
zlib@1.3.2: commit resolved from devbox, 2 patches applied, out, dev, and static outputs.
openssl@3.6.0: ELF mapping with the debug output, 38 of 40 debug files mapped to their source.

Navigation Menu

nixpkgs-clarity: Design and implement Scancode pipeline for single nixpkg #1938

67 remaining items

Summary

Steps

Out of scope for v1

Questions to settle

Input

Steps

Installation

Tested on

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

nixpkgs-clarity: Design and implement Scancode pipeline for single nixpkg #1938

Description

Activity

pombredanne commented on Mar 6, 2026

chinyeungli commented on May 29, 2026

chinyeungli commented on Jul 22, 2026

chinyeungli commented on Aug 18, 2026

67 remaining items

chinyeungli commented on Nov 4, 2026

Summary

Steps

Out of scope for v1

Questions to settle

tdruez commented on Nov 5, 2026

Input

Steps

Installation

Tested on

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions