System resource cache by JonoYang · Pull Request #5289 · aboutcode-org/scancode-toolkit

JonoYang · 2026-08-25T21:45:03Z

This PR introduces a cache for the results of scanners run by scancode. The scancode, scan_codebase, and scan_resource functions have been updated to look for cached scan results to reuse before scanning. The cache is updated after scanning, if it was required.

This is related to #3941 #4556

Signed-off-by: Jono Yang <jyang@nexb.com>

* Use sha256 instead of md5 Signed-off-by: Jono Yang <jyang@nexb.com>

Signed-off-by: Jono Yang <jyang@nexb.com>

* Create envvar for controlling whether or not the scan results cache is used Signed-off-by: Jono Yang <jyang@nexb.com>

Signed-off-by: Jono Yang <jyang@nexb.com>

* The runner cannot instantiate a cryptography library of some kind Signed-off-by: Jono Yang <jyang@nexb.com>

AyanSinhaMahapatra

@JonoYang LGTM! Thanks++
See some of my comments for your consideration.
I also did some basic scans which shows negligible performance impact on cache creation and significant performance gain from using cached results. But we should probably do some proper benchmarks to test the performance gains too.

The archives I used for running the benchmarks:

First scan of v2.1.2 without --use-cached-results:

 scancode -clipeu syncthing-2.1.2.tar.gz-extract/  --json-pp syncthing-2.1.2-clipeu.json -n 6
Setup plugins...
Collect file inventory...
Scan files for: info, packages, licenses, copyrights, emails, urls with 6 process(es)...
[####################] 1868
Scanning done.
Summary:        info, packages, licenses, copyrights, emails, urls with 6 process(es)
Errors count:   0
Scan Speed:     7.71 files/sec. 110.88 KB/sec.--use-cached-results
Initial counts: 1133 resource(s): 934 file(s) and 199 directorie(s)
Final counts:   1133 resource(s): 934 file(s) and 199 directorie(s) for 13.11 MB
Timings:
  scan_start: 2026-09-07T100805.133779
  scan_end:   2026-09-07T101008.901230
  setup_scan:licenses: 2.45s
  setup: 2.45s
  inventory: 0.11s
  scan:packages: 0.13s
  scan: 121.10s
  output:json-pp: 0.34s
  output: 0.34s
  total: 124.13s
Removing temporary files...done.

First scan of v2.1.3 without --use-cached-results:

scancode -clipeu syncthing-2.1.3.tar.gz-extract/  --json-pp syncthing-2.1.3-clipeu.json -n 6
Setup plugins...
Collect file inventory...
Scan files for: info, packages, licenses, copyrights, emails, urls with 6 process(es)...
[####################] 1866
Scanning done.
Summary:        info, packages, licenses, copyrights, emails, urls with 6 process(es)
Errors count:   0
Scan Speed:     7.40 files/sec. 106.58 KB/sec.
Initial counts: 1132 resource(s): 933 file(s) and 199 directorie(s)
Final counts:   1132 resource(s): 933 file(s) and 199 directorie(s) for 13.12 MB
Timings:
  scan_start: 2026-09-07T101019.832652
  scan_end:   2026-09-07T101228.601000
  setup_scan:licenses: 2.47s
  setup: 2.47s
  inventory: 0.11s
  scan:packages: 0.11s
  scan: 126.07s
  output:json-pp: 0.32s
  output: 0.32s
  total: 129.11s

Scan of v2.1.2 with --use-cached-results (only creating cache):

scancode -clipeu syncthing-2.1.2.tar.gz-extract/  --json-pp syncthing-2.1.2-clipeu.json -n 6 --use-cached-results
Setup plugins...
Collect file inventory...
Scan files for: info, packages, licenses, copyrights, emails, urls with 6 process(es)...
[####################] 1868
Scanning done.
Summary:        info, packages, licenses, copyrights, emails, urls with 6 process(es)
Errors count:   0
Scan Speed:     7.70 files/sec. 110.63 KB/sec.
Initial counts: 1133 resource(s): 934 file(s) and 199 directorie(s)
Final counts:   1133 resource(s): 934 file(s) and 199 directorie(s) for 13.11 MB
Timings:
  scan_start: 2026-09-07T101433.426523
  scan_end:   2026-09-07T101637.529302
  setup_scan:licenses: 2.49s
  setup: 2.49s
  inventory: 0.12s
  scan:packages: 0.12s
  scan: 121.38s
  output:json-pp: 0.34s
  output: 0.34s
  total: 124.46s
Removing temporary files...done.

Scan of v2.1.3 with --use-cached-results (using cached results on another version with highly similar files):

scancode -clipeu syncthing-2.1.3.tar.gz-extract/  --json-pp syncthing-2.1.3-clipeu.json -n 6 --use-cached-results
Setup plugins...
Collect file inventory...
Scan files for: info, packages, licenses, copyrights, emails, urls with 6 process(es)...
[####################] 1866
Scanning done.
Summary:        info, packages, licenses, copyrights, emails, urls with 6 process(es)
Errors count:   0
Scan Speed:     45.50 files/sec. 655.31 KB/sec.
Initial counts: 1132 resource(s): 933 file(s) and 199 directorie(s)
Final counts:   1132 resource(s): 933 file(s) and 199 directorie(s) for 13.12 MB
Timings:
  scan_start: 2026-09-07T102406.378978
  scan_end:   2026-09-07T102429.571458
  setup_scan:licenses: 2.45s
  setup: 2.46s
  inventory: 0.11s
  scan:packages: 0.13s
  scan: 20.50s
  output:json-pp: 0.33s
  output: 0.33s
  total: 23.55s
Removing temporary files...done.

Scan of v2.1.2 with --use-cached-results (using cached results on a same version with exactly similar files):

scancode -clipeu syncthing-2.1.2.tar.gz-extract/  --json-pp syncthing-2.1.2-clipeu.json -n 6 --use-cached-results
Setup plugins...
Collect file inventory...
Scan files for: info, packages, licenses, copyrights, emails, urls with 6 process(es)...
[####################] 1868
Scanning done.
Summary:        info, packages, licenses, copyrights, emails, urls with 6 process(es)
Errors count:   0
Scan Speed:     1563.51 files/sec. 21.95 MB/sec.
Initial counts: 1133 resource(s): 934 file(s) and 199 directorie(s)
Final counts:   1133 resource(s): 934 file(s) and 199 directorie(s) for 13.11 MB
Timings:
  scan_start: 2026-09-07T102440.089470
  scan_end:   2026-09-07T102443.410571
  setup_scan:licenses: 2.50s
  setup: 2.50s
  inventory: 0.11s
  scan:packages: 0.13s
  scan: 0.60s
  output:json-pp: 0.33s
  output: 0.33s
  total: 3.67s
Removing temporary files...done.

Summary:

scan of v2.1.2: 124.13s
scan of v2.1.3: 129.11s
scan of v2.1.2 only creating cache: 124.46s (negligible difference)
scan of v2.1.3, using cached results on another version with highly similar files: 23.55s
scan of v2.1.2, using cached results on a same version with exactly similar files: 3.67s

AyanSinhaMahapatra · 2026-09-07T09:42:07Z

+    return hasher
+
+
+def compute_results_cache_index(location, filename):


Can this be reused from functions in commoncode.hash? The code there was updated to work on chunks a while ago too.

AyanSinhaMahapatra · 2026-09-07T09:46:18Z

+
+def get_results_cache_directory_location(results_cache_index, results_cache_dir=results_cache_dir):
+    """
+    Return the location of the directory containing the cache files for a given


Should we include an example results_cache_index string in the docstring?

AyanSinhaMahapatra · 2026-09-07T09:54:45Z

+    When run as a stand-alone release, the results cache is tied to specific
+    ScanCode toolkit versions. When run in development mode, the cache is stored
+    in the `scancode-toolkit/.cache` directory.
+


Should we include some functionality to re-compute the cache for a file based on data structure changes caused by different versions of scancode, or different CLI options used with different output data formats? There is some discussion on this at #3941

We need to check that we atleast handle the errors resulting out of this. Additionally it would be nice to somehow include the data format versions in the index cache too, smartly avoiding cache recompute whenever possible.

Output data formats are not always changes based on scancode versions, and an update in the output data format version also doesn't correspond to updates in all the scanner output data. So we need to probably compute a hash based on all the data fields present for a particular scanner data based on the keys present in the mapping and include this in the index file path? This would cover both variations caused by scancode version changes and CLI options used.

This is an enhancement though, so feel free to mark this out of scope.

AyanSinhaMahapatra · 2026-09-07T09:55:57Z

+@click.option('--use-cached-results',
+    is_flag=True,
+    default=False,
+    hidden=True,


We could leave this as-is and not hidden, as there is an EXPERIMENTAL tag in the help text already. We want more people to see and test out the functionality probably, what do you think?

AyanSinhaMahapatra · 2026-09-07T10:18:52Z

    hidden=True,
    help_group=cliutils.MISC_GROUP, sort_order=1000, cls=PluggableCommandLineOption)
+
+@click.option('--use-cached-results',


Was just wondering if this is the best name since this option enables both the creation of the cache and the use of cached results and this name only indicates the latter.

Alternatives could be: --use-results-cache, --cache-results, --cache but I'm not sure what to choose too.

One other option is to use two other options one to create the cache and the other to use the cached results, but using just one option is probably much better if there are no large performance implications.

AyanSinhaMahapatra · 2026-09-07T10:19:54Z

+    is_flag=True,
+    default=False,
+    hidden=True,
+    help='(EXPERIMENTAL) ScanCode will use cached results during scan time.',


We need to highlight in the help text also that this option both:

caches scan results from scans

uses cached scan results

github-advanced-security Bot found potential problems Aug 25, 2026

View reviewed changes

Comment thread src/scancode/results_cache.py Dismissed

Comment thread src/scancode/results_cache.py Dismissed

Comment thread src/scancode/results_cache.py Dismissed

Comment thread src/scancode/results_cache.py Dismissed

github-advanced-security Bot found potential problems Aug 28, 2026

View reviewed changes

Comment thread tests/scancode/test_results_cache.py Dismissed

JonoYang added 13 commits August 28, 2026 16:41

Create plugin for resource cache indices

35f63cc

Signed-off-by: Jono Yang <jyang@nexb.com>

Use pypi hash strategy for organizing resource cache

c54d124

* Use sha256 instead of md5 Signed-off-by: Jono Yang <jyang@nexb.com>

Place resource cache into scancode cache dir

212d559

Signed-off-by: Jono Yang <jyang@nexb.com>

Update scan_resource to get and save data to cache

e4fde9a

Signed-off-by: Jono Yang <jyang@nexb.com>

Create directories before opening or creating cache files

0916e64

* Create envvar for controlling whether or not the scan results cache is used Signed-off-by: Jono Yang <jyang@nexb.com>

Rename resource_cache_index to results_cache_index

d69dfc7

Signed-off-by: Jono Yang <jyang@nexb.com>

Change option no-cached-results to use-cached-results

fc0856a

Signed-off-by: Jono Yang <jyang@nexb.com>

Fix logic where we run scanners after getting cached results

866309a

Signed-off-by: Jono Yang <jyang@nexb.com>

Create test for use-cached-results cli option

7988a13

Signed-off-by: Jono Yang <jyang@nexb.com>

Create results_cache_index from filename instead of path

fa06544

Signed-off-by: Jono Yang <jyang@nexb.com>

Update CHANGELOG.rst

5af2be0

Signed-off-by: Jono Yang <jyang@nexb.com>

Create docstring for results_cache module

630074d

Signed-off-by: Jono Yang <jyang@nexb.com>

Create test for results_cache.py

60ee685

Signed-off-by: Jono Yang <jyang@nexb.com>

JonoYang force-pushed the system-resource-cache branch from a1aa058 to 60ee685 Compare August 28, 2026 23:43

JonoYang added 2 commits August 31, 2026 16:32

Update cli-core-options.rst #3941 #5289

ab1932a

Signed-off-by: Jono Yang <jyang@nexb.com>

Set PYTHONHASHSEED for windows test #3941 #4556

1247f39

Signed-off-by: Jono Yang <jyang@nexb.com>

JonoYang force-pushed the system-resource-cache branch from e965307 to 1247f39 Compare September 3, 2026 23:25

Skip results cache test on windows #3941 #4556

eb13ccb

* The runner cannot instantiate a cryptography library of some kind Signed-off-by: Jono Yang <jyang@nexb.com>

JonoYang requested a review from AyanSinhaMahapatra September 4, 2026 00:53

JonoYang mentioned this pull request Sep 5, 2026

swh-cc: Create code to efficiently run scans #4556

Open

AyanSinhaMahapatra approved these changes Sep 7, 2026

View reviewed changes

Navigation Menu

Uh oh!

Uh oh!

System resource cache - #5289

System resource cache#5289
JonoYang wants to merge 16 commits into
developfrom
system-resource-cache

JonoYang commented Aug 25, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AyanSinhaMahapatra left a comment •

edited

Loading

Uh oh!

AyanSinhaMahapatra Sep 7, 2026

Uh oh!

AyanSinhaMahapatra Sep 7, 2026

Uh oh!

AyanSinhaMahapatra Sep 7, 2026

Uh oh!

AyanSinhaMahapatra Sep 7, 2026

Uh oh!

AyanSinhaMahapatra Sep 7, 2026

Uh oh!

AyanSinhaMahapatra Sep 7, 2026

Uh oh!

AyanSinhaMahapatra Sep 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		return hasher


		def compute_results_cache_index(location, filename):

Uh oh!

Uh oh!

Conversation

JonoYang commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AyanSinhaMahapatra left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AyanSinhaMahapatra Sep 7, 2026

Choose a reason for hiding this comment

Uh oh!

AyanSinhaMahapatra Sep 7, 2026

Choose a reason for hiding this comment

Uh oh!

AyanSinhaMahapatra Sep 7, 2026

Choose a reason for hiding this comment

Uh oh!

AyanSinhaMahapatra Sep 7, 2026

Choose a reason for hiding this comment

Uh oh!

AyanSinhaMahapatra Sep 7, 2026

Choose a reason for hiding this comment

Uh oh!

AyanSinhaMahapatra Sep 7, 2026

Choose a reason for hiding this comment

Uh oh!

AyanSinhaMahapatra Sep 7, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

JonoYang commented Aug 25, 2026 •

edited

Loading

AyanSinhaMahapatra left a comment •

edited

Loading