diff --git a/docs/faq.rst b/docs/faq.rst index ee5a43ba8d..fcfdbbb012 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -582,7 +582,7 @@ If you create a new keyfile-encrypted repository at the same filesystem path multiple times (for example, when a previous repository at that path was moved away or unmounted), Borg will not overwrite or reuse an existing key file in your keys directory. Instead, each repository gets a key file -of its own, named after the SHA-256 hash of the key file's own content -- +of its own, named after the BLAKE3 hash of the key file's own content -- the header line naming the repository's ID, followed by the encrypted key material. Both the repository ID and the key material are freshly randomized when the repository is created, so two repositories, even ones @@ -598,7 +598,7 @@ names shortened for readability): Each belongs to a distinct repository, wherever it was created -- a name collision between different repositories would require an outright -SHA-256 hash collision, not just an unlucky path reuse. Borg does not use +BLAKE3 hash collision, not just an unlucky path reuse. Borg does not use the key file name to find the right key either: to open a repository, it scans all files in the keys directory (see :ref:`env_vars` for ``BORG_KEYS_DIR``) and picks the one whose header names that repository's diff --git a/docs/internals/data-structures.rst b/docs/internals/data-structures.rst index ecb726aee9..c51bb6347f 100644 --- a/docs/internals/data-structures.rst +++ b/docs/internals/data-structures.rst @@ -50,7 +50,7 @@ archives/ The (encrypted and compressed) repository objects are not stored one store object each: many of them are batched into a **pack** file and that pack file -is stored as a single store object. Its name is the hex-encoded sha256 hash of +is stored as a single store object. Its name is the hex-encoded blake3 hash of the pack file's content: packs/ @@ -61,7 +61,7 @@ index/ 0000... .. ffff... the chunks index (chunk ID -> location within a pack file), stored as a set of immutable, encrypted index fragments. A fragment's name is the - hex-encoded sha256 hash of its content. + hex-encoded blake3 hash of its content. See :ref:`packs` for the pack file format, the ``index/`` namespace and how both are written and compacted. @@ -77,7 +77,7 @@ cache/ check finishes. referenced-by-archive. what one archive references (object ID -> plaintext object size), plus the file - count and content size of that archive, with an appended sha256 for integrity. + count and content size of that archive, with an appended blake3 hash for integrity. It lets a following ``borg compact`` or ``borg analyze`` skip re-reading the items of an unchanged archive. chunkindex-invalid @@ -92,7 +92,7 @@ all clients); it is not the client-local cache described in keys/ When using repokey mode, the encrypted, passphrase protected borg keys are - stored here as a base64 encoded text. The sha256 content hash of the + stored here as a base64 encoded text. The blake3 content hash of the stored borg key is used for the name. A repository may contain *multiple* such borg keys (one per passphrase) to @@ -724,7 +724,7 @@ The files cache The **files cache** is a client-local file, stored in the borg cache directory of the repository (see :ref:`env_vars`) as ``files.``. SUFFIX is the -sha256 of the archive (series) name, so each archive series gets its own files +blake3 hash of the archive (series) name, so each archive series gets its own files cache; ``BORG_FILES_CACHE_SUFFIX`` overrides it. The files cache is used at backup time to quickly determine whether a given file is unchanged and we have all its chunks. @@ -1116,7 +1116,7 @@ All modes Encryption keys (and other secrets) are kept either in the keys directory on the client ('keyfile' mode) or under the keys/ namespace in the repository -('repokey' mode) using the sha256 of the borg key content as the name. +('repokey' mode) using the blake3 hash of the borg key content as the name. In both cases, the secrets are generated from random and then encrypted by a key derived from your passphrase (this happens on the client before the key diff --git a/docs/internals/packs.rst b/docs/internals/packs.rst index 5f97d22f34..4800e8c88a 100644 --- a/docs/internals/packs.rst +++ b/docs/internals/packs.rst @@ -120,7 +120,7 @@ blobs after the damaged one are still found; the damaged blob itself is dropped, it can not be read back. The walk rebuilds the index from the pack as it is: the damaged bytes stay where -they are, as a gap no index entry covers. A pack is named by the sha256 of its +they are, as a gap no index entry covers. A pack is named by the blake3 hash of its content, so a pack damaged in the store keeps failing the store-level check that ``borg check`` runs over ``packs/``, also after ``borg check --repair`` has rebuilt the index from it. Rewriting such a pack is repository-level repair, see @@ -161,9 +161,9 @@ Blobs follow one another contiguously with no padding:: Pack ID ~~~~~~~ -The pack ID is the SHA-256 of the pack file's bytes:: +The pack ID is the 256 bit BLAKE3 hash of the pack file's bytes:: - pack_id = sha256(pack_bytes) + pack_id = blake3(pack_bytes) Content-addressing the file by its own bytes makes the name commit to the content, so borgstore can verify and cache it and ``borg check`` can detect @@ -258,10 +258,10 @@ A fragment is a serialized ``ChunkIndex`` (a ``borghash`` ``HashTableNT`` keyed of each entry are zeroed before serializing. Fragments are **not** encrypted: they map ``chunk_id`` to ``(pack_id, obj_offset, obj_size)``, which anyone with access to the repository could equally well read out of the unencrypted blob headers (see -:ref:`pack-recovery`). A fragment's name is the SHA-256 digest of its own content:: +:ref:`pack-recovery`). A fragment's name is the BLAKE3 digest of its own content:: index/ - + An ordinary backup writes only the entries that are new in that session; a full rewrite (e.g. by ``borg compact``) writes all of them. In both cases the write is diff --git a/docs/internals/security.rst b/docs/internals/security.rst index 583cfa1dcd..66c2b6e3d0 100644 --- a/docs/internals/security.rst +++ b/docs/internals/security.rst @@ -309,7 +309,7 @@ The ciphertext is then converted to base64. This base64-encoded *borg key* is then stored in the key file or under the repository's ``keys/`` namespace (keyfile and repokey modes respectively), named -by the sha256 of its content. +by the blake3 hash of its content. The use of a constant IV is secure because an identical passphrase will result in a different derived KEK for every key encryption due to the salt. @@ -325,7 +325,7 @@ key material. This lets several people access a shared repository with independent passphrases, without sharing one secret. Or you can add borg keys for redundant, more fault-tolerant storage. -keyfile and repokey borg keys use the same format and the same sha256-content +keyfile and repokey borg keys use the same format and the same blake3-content naming; borg locates a borg key independently of its key type byte and tries each available one against the supplied passphrase until one decrypts. A borg key may carry a label for management. The constant-IV argument above still holds, because @@ -377,7 +377,7 @@ used: object's metadata slot and data slot are encrypted and authenticated with the borg key (see :ref:`security_encryption`); its per-object header is unencrypted and carries the magic, the format version and the chunk id (see :ref:`pack-format`). -- ``index/`` -- the chunk id to pack location index. It is not encrypted, +- ``index/`` -- the chunk id to pack location index. It is not encrypted, but it only contains chunk ids and locations, which the pack headers expose anyway. - ``archives/`` -- one empty object per archive. The archive name, its timestamps, the item metadata and the chunk lists all live inside encrypted @@ -386,7 +386,7 @@ used: modification time. - ``config/manifest`` (an encrypted repository object), plus the plaintext ``config/version``, ``config/id`` and ``config/readme``. -- ``keys/`` -- in ``repokey`` mode, the borg key(s), encrypted with the +- ``keys/`` -- in ``repokey`` mode, the borg key(s), encrypted with the passphrase-derived KEK (see :ref:`key_encryption`). - ``locks/*`` and ``cache/*``. Note that the per-archive reference caches ``cache/referenced-by-archive.``, written by ``borg compact`` and diff --git a/docs/usage/key.rst b/docs/usage/key.rst index 2b29d402dd..24f222341d 100644 --- a/docs/usage/key.rst +++ b/docs/usage/key.rst @@ -31,7 +31,7 @@ Examples .. note:: - Automatically placed key files are named after the SHA-256 hash of their own + Automatically placed key files are named after the BLAKE3 hash of their own contents, not after the repository directory name. Because changing the passphrase re-encrypts the key, the key file is rewritten under a new name and the previous one is removed — that is why the two paths above differ. Use diff --git a/src/borg/archive.py b/src/borg/archive.py index c30a24b6b7..0a8174ed1d 100644 --- a/src/borg/archive.py +++ b/src/borg/archive.py @@ -2203,7 +2203,7 @@ def check( self.format = format self.repository = repository # A normal (non-repair) archives check trusts the in-repo index: the repository check verified - # each index object's sha256, and the index is the authoritative record of which chunks exist, + # each index object's blake3 hash, and the index is the authoritative record of which chunks exist, # so we do not rebuild it from the packs (reading every pack is far too slow for a routine check). # --repair does rebuild from the packs (slow_rebuild=repair), working from the real packs so it # can detect and fix archives that reference chunks whose pack has gone missing. diff --git a/src/borg/archiver/check_cmd.py b/src/borg/archiver/check_cmd.py index abbffb92de..b5231c1d58 100644 --- a/src/borg/archiver/check_cmd.py +++ b/src/borg/archiver/check_cmd.py @@ -112,12 +112,12 @@ def build_parser_check(self, subparsers, common_parser, mid_common_parser): It consists of two major steps: 1. Checking the consistency of the repository itself. The objects in the ``index/`` - and ``packs/`` namespaces are named by the sha256 hash of their content, so such + and ``packs/`` namespaces are named by the blake3 hash of their content, so such an object is intact if and only if the hash of its content still equals its name. The check verifies the (small) index objects first and, only if they are intact, all packs. It also cross-checks the chunk index against the packs present in the repository to detect referenced but missing packs. Bit rot and other types of - accidental damage can be detected this way, but as sha256 content-addressing is + accidental damage can be detected this way, but as content-addressing is not a MAC, this step does not detect tampering. Running the repository check can be split into multiple partial checks using ``--max-duration``. For rest:// repositories, the server computes the hashes, so the pack contents do diff --git a/src/borg/cache.py b/src/borg/cache.py index bf1686e472..5ebee35c61 100644 --- a/src/borg/cache.py +++ b/src/borg/cache.py @@ -1,5 +1,4 @@ import configparser -import hashlib import io import os import shutil @@ -38,6 +37,7 @@ from .helpers.msgpack import int_to_timestamp, timestamp_to_int from .item import ChunkListEntry from .crypto.file_integrity import IntegrityCheckedFile, FileIntegrityError +from .crypto.key import blake3_256, blake3_256_hex from .manifest import Manifest from .platform import SaveFile from .repository import Repository, StoreObjectNotFound, PackReader @@ -57,7 +57,7 @@ def files_cache_name(archive_name, files_cache_name="files"): # when not, the user may manually do that by using the env var. if not suffix: # avoid issues with too complex or long archive_name by hashing it: - suffix = hashlib.sha256(archive_name.encode()).hexdigest() + suffix = blake3_256_hex(archive_name.encode()) return files_cache_name + "." + suffix @@ -595,7 +595,7 @@ def list_chunkindex_fragments(repository): """List the index/ fragments, returning each fragment's (name, approximate entry count). This is the single primitive that walks the index/ namespace; list_chunkindex_hashes is a thin - wrapper over it. In that namespace each object's name is the sha256 hash of its content. The entry + wrapper over it. In that namespace each object's name is the blake3 hash of its content. The entry count is estimated from the stored object's byte size (chunkindex_fragment_entry_size() bytes per entry), so we can classify fragments (small vs. sealed) without loading them. The estimate ignores the small fixed header, which is negligible for the fragment sizes we care about. @@ -663,10 +663,10 @@ def delete_chunkindex_from_repo(repository): def _store_chunkindex_fragment(repository, batch, stored_hashes, *, force_write): - """Serialize a temporary ChunkIndex `batch` and store it as an index/ fragment. + """Serialize a temporary ChunkIndex `batch` and store it as an index/ fragment. We don't serialize the flags or the size, so callers pass entries with those zeroed. The object - is stored under index/, where is the sha256 of its content, so borgstore can verify + is stored under index/, where is the blake3 hash of its content, so borgstore can verify it like any other object; an incompatible format from a different borg version is rejected by borghash's own versioned header (MAGIC + VERSION) when read back. @@ -676,7 +676,7 @@ def _store_chunkindex_fragment(repository, batch, stored_hashes, *, force_write) with io.BytesIO() as f: batch.write(f) data = f.getvalue() - new_hash = hashlib.sha256(data).hexdigest() + new_hash = blake3_256_hex(data) stored = False if force_write or new_hash not in stored_hashes: index_name = f"index/{new_hash}" @@ -820,7 +820,7 @@ def read_chunkindex_from_repo(repository, hash): except StoreObjectNotFound: logger.debug(f"{index_name} not found in the repository.") else: - if hashlib.sha256(chunks_data).digest() == hex_to_bin(hash): + if blake3_256(chunks_data) == hex_to_bin(hash): logger.debug(f"{index_name} is valid.") try: with io.BytesIO(chunks_data) as f: @@ -1050,7 +1050,7 @@ def build_chunkindex_from_repo( # cache/referenced-by-archive.. it lets a following compact or analyze skip re-scanning # an unchanged archive's items. the blob is: file_count (uint64 LE), content_size (uint64 LE), a # serialized HashTableNT mapping object id (32 bytes) -> plaintext object size (uint32), and a -# sha256 of all of that appended for integrity. +# blake3 hash of all of that appended for integrity. REFERENCED_BY_ARCHIVE = "referenced-by-archive." # name prefix within the "cache" store namespace ArchiveReferenceEntry = namedtuple("ArchiveReferenceEntry", "size") ArchiveReferenceEntryFormatT = namedtuple("ArchiveReferenceEntryFormatT", "size") @@ -1081,11 +1081,11 @@ def load_archive_references(repository, archive_id: bytes): data = repository.store_load(archive_reference_cache_name(archive_id)) except StoreObjectNotFound: return None - # the serialized blob has a sha256 of its content appended (the store name cannot also carry it, - # as borgstore's name length limit is too small for archive id hex + sha256 hex). a mismatch means + # the serialized blob has a blake3 hash of its content appended (the store name cannot also carry + # it, as borgstore's name length limit is too small for archive id hex + hash hex). a mismatch means # the cache is corrupted; we then return None so the caller falls back to scanning the archive. hex_id = bin_to_hex(archive_id) - if len(data) < 16 + 32 or hashlib.sha256(data[:-32]).digest() != data[-32:]: + if len(data) < 16 + 32 or blake3_256(data[:-32]) != data[-32:]: logger.warning(f"Ignoring corrupted references cache of archive {hex_id}.") return None try: @@ -1100,13 +1100,13 @@ def load_archive_references(repository, archive_id: bytes): def store_archive_references(repository, archive_id: bytes, references) -> None: - """Serialize the references (a small header plus the id->size table, with a sha256 appended).""" + """Serialize the references (a small header plus the id->size table, with a blake3 hash appended).""" with io.BytesIO() as f: f.write(references.file_count.to_bytes(8, "little")) f.write(references.content_size.to_bytes(8, "little")) references.ids.write(f) data = f.getvalue() - data += hashlib.sha256(data).digest() + data += blake3_256(data) repository.store_store(archive_reference_cache_name(archive_id), data) diff --git a/src/borg/constants.py b/src/borg/constants.py index edb4a440d3..a071072ba2 100644 --- a/src/borg/constants.py +++ b/src/borg/constants.py @@ -129,7 +129,7 @@ # repo.list() result count limit used by the Borg client LIST_SCAN_LIMIT = 100000 -# The chunks index is stored in the repo as immutable, content-addressed index/ fragments. +# The chunks index is stored in the repo as immutable, content-addressed index/ fragments. # We keep each fragment's entry count within [MIN, MAX] where possible: MAX bounds a fragment's size, # MIN keeps the fragment count down. Small (< MIN) fragments are merged (repacked); fragments already # in range are left untouched (stable/immutable). SMALL_FRAGMENT_CAP bounds how many sub-MIN fragments diff --git a/src/borg/crypto/key.py b/src/borg/crypto/key.py index 23a5f8fa63..4561457171 100644 --- a/src/borg/crypto/key.py +++ b/src/borg/crypto/key.py @@ -35,7 +35,7 @@ def keyfile_name_for(content: bytes) -> str: - return sha256(content).hexdigest() + return blake3_256_hex(content) KEYFILE_ID = "BORG_KEY" @@ -87,6 +87,27 @@ def get_blake3_mt_threshold() -> int: return _blake3_mt_threshold +def _blake3_hasher(data: bytes): + # big inputs (packs, index fragments) are hashed multi-threaded from get_blake3_mt_threshold() + # on; the hash is the same either way. + max_threads = blake3.AUTO if len(data) >= get_blake3_mt_threshold() else 1 + return blake3(data, max_threads=max_threads) + + +def blake3_256(data: bytes) -> bytes: + """Return the unkeyed 256 bit blake3 hash of *data* (32 bytes).""" + return _blake3_hasher(data).digest(length=32) + + +def blake3_256_hex(data: bytes) -> str: + """Return the unkeyed 256 bit blake3 hash of *data* as 64 lowercase hex digits. + + Content-addressed store objects (packs/, index/, keys/, locks/) and automatically named + keyfiles are named by this hash of their content. + """ + return _blake3_hasher(data).hexdigest(length=32) + + def is_keyfile(data: str | bytes, repoid: str | None = None) -> bool: # repoid is a hex str, if given. if given, we only accept keyfiles for that repo. header = f"{KEYFILE_ID} {repoid or ''}" @@ -803,7 +824,7 @@ def _repo_candidates(self): else: keydata = repo.load_key() if keydata: - result.append((sha256(keydata).hexdigest(), keydata.decode("utf-8"), None)) + result.append((blake3_256_hex(keydata), keydata.decode("utf-8"), None)) return result def _keyfile_candidates(self): @@ -816,7 +837,7 @@ def _keyfile_candidates(self): blob = fd.read() except OSError: continue - result.append((sha256(blob).hexdigest(), blob.decode("utf-8"), str(path))) + result.append((blake3_256_hex(blob), blob.decode("utf-8"), str(path))) return result def _iter_keys(self): @@ -883,7 +904,7 @@ def load(self, target, passphrase): blob = fd.read() except OSError: return False - return self._try_key(sha256(blob).hexdigest(), blob.decode("utf-8"), str(target), passphrase) + return self._try_key(blake3_256_hex(blob), blob.decode("utf-8"), str(target), passphrase) else: return self.load_any(passphrase) @@ -918,7 +939,7 @@ def save(self, target, passphrase, algorithm, create=False, label=None, replace= secure_erase(old_target, avoid_collateral_damage=True) except OSError as exc: logger.debug('Could not remove previous keyfile "%s": %s', old_target, exc) - self._loaded_key_id = sha256(keyfile_data.encode()).hexdigest() + self._loaded_key_id = blake3_256_hex(keyfile_data.encode()) elif self.storage == KeyBlobStorage.REPO: self.logically_encrypted = passphrase != "" # nosec B105 key_data = keyfile_format(bin_to_hex(self.repository_id), key_data) @@ -929,7 +950,7 @@ def save(self, target, passphrase, algorithm, create=False, label=None, replace= self._loaded_key_id = store_key(key_data) else: target.save_key(key_data) # legacy repository: single borg key - self._loaded_key_id = sha256(key_data).hexdigest() + self._loaded_key_id = blake3_256_hex(key_data) else: raise TypeError("Unsupported borg key storage type") self.target = target if self.storage != KeyBlobStorage.REPO else self.repository @@ -1161,9 +1182,7 @@ class Blake3ChecksumKey(ChecksumKeyBase): IDHASH_NAME = "blake3" def id_hash(self, data): - # see ID_BLAKE3_256.id_hash about max_threads - max_threads = blake3.AUTO if len(data) >= get_blake3_mt_threshold() else 1 - return blake3(data, max_threads=max_threads).digest(length=32) + return blake3_256(data) def mac(self, prefix, payload): max_threads = blake3.AUTO if len(payload) >= get_blake3_mt_threshold() else 1 diff --git a/src/borg/repository.py b/src/borg/repository.py index fa1fb6b495..700199d935 100644 --- a/src/borg/repository.py +++ b/src/borg/repository.py @@ -6,7 +6,6 @@ import time from collections import defaultdict, namedtuple from pathlib import Path -from hashlib import sha256 from borghash import HashTableNT @@ -31,13 +30,19 @@ from .logger import create_logger from .manifest import NoManifestError from .repoobj import RepoObj, OBJ_MAGIC, SUPPORTED_OBJ_VERSIONS -from .crypto.key import is_keyfile +from .crypto.key import is_keyfile, blake3_256, blake3_256_hex logger = create_logger(__name__) -# an object name is its sha256 as 64 lowercase hex digits. +# an object name is the hex blake3 hash of the object's content (64 lowercase hex digits), see +# crypto.key.blake3_256_hex(). _valid_object_name = re.compile(r"[0-9a-f]{64}").fullmatch +# the hash algorithm content-addressed objects (packs/, index/) are named by, as borgstore calls it: +# borgstore names the packs it writes for us (defrag) and verifies objects (hash) with it, so it must +# be one borgstore supports, and it must match blake3_256(). +NAME_HASH = "blake3" + # how much of a pack PackReader reads at once when searching for the next object header. RESYNC_WINDOW_SIZE = 1024 * 1024 # how much to read to get an object's header plus, at the usual metadata slot sizes, its metadata @@ -246,10 +251,10 @@ def _store_pieces(self, pieces, outcome, trace=False): # that incremental string concatenation would cause in Python). pack_data = b"".join(cdata for _, cdata in pieces) - # Name the pack by the SHA-256 of its bytes: the name commits to the stored content, + # Name the pack by the blake3 hash of its bytes: the name commits to the stored content, # so borgstore can verify and cache the file. self._trace("H", trace) - pack_id = sha256(pack_data).digest() + pack_id = blake3_256(pack_data) # Record (chunk_id, pack_id, obj_offset, obj_size) for every piece. results = [] @@ -595,13 +600,13 @@ class PackTracker: Records are kept across checks: intact records (result=1) are reused by checks run with max_age, corrupt records (result=0) are kept for repair and always re-verified. Records of packs no longer listed in packs/ are pruned when a check finishes scanning packs/. - Stored at cache/checked-packs as the serialized table with a sha256 over it appended. + Stored at cache/checked-packs as the serialized table with a blake3 hash over it appended. new() starts an empty tracker, load() reads the stored one. """ NAME = "cache/checked-packs" KEY_SIZE = 32 # pack id - DIGEST_SIZE = 32 # sha256 + DIGEST_SIZE = 32 # blake3_256 Entry = namedtuple("Entry", "timestamp result") EntryFormatT = namedtuple("EntryFormatT", "timestamp result") _EntryFormat = EntryFormatT(timestamp="Q", result="B") # unix ts, 1=ok 0=corrupt @@ -620,14 +625,14 @@ def new(cls, store): def load(cls, store): """Return a tracker holding the stored table. - Return an empty one if cache/checked-packs is missing, its appended sha256 does not match, + Return an empty one if cache/checked-packs is missing, its appended blake3 hash does not match, it does not deserialize, or its entries do not have this class's key size and Entry layout. """ try: data = store.load(cls.NAME) except StoreObjectNotFound: return cls.new(store) - if len(data) < cls.DIGEST_SIZE or sha256(data[: -cls.DIGEST_SIZE]).digest() != data[-cls.DIGEST_SIZE :]: + if len(data) < cls.DIGEST_SIZE or blake3_256(data[: -cls.DIGEST_SIZE]) != data[-cls.DIGEST_SIZE :]: logger.warning("Ignoring corrupted checked-packs set.") return cls.new(store) try: @@ -676,7 +681,7 @@ def save(self): with io.BytesIO() as f: self.table.write(f) data = f.getvalue() - self.store.store(self.NAME, data + sha256(data).digest()) + self.store.store(self.NAME, data + blake3_256(data)) def clear(self): self.table.clear() @@ -941,7 +946,7 @@ def store_key(self, keydata): # store a single repokey borg key (content-addressed). does NOT delete other borg keys, # so a repository can have multiple borg keys (one per passphrase). returns the # store object name (= borg key id) under which the borg key was stored. - digest = sha256(keydata).hexdigest() + digest = blake3_256_hex(keydata) self.store.store(f"keys/{digest}", keydata) return digest @@ -1126,9 +1131,9 @@ def info(self): def check(self, repair=False, max_duration=0, max_age=0, repo_only=False): """Check repository consistency. - packs/ and index/ objects are named by the sha256 of their content, so a pack or index file - is intact iff store.hash(name) still equals name. The whole pack is hashed; the REST backend - computes the hash server-side, so for it nothing is downloaded. + packs/ and index/ objects are named by the blake3 hash of their content, so a pack or index + file is intact iff store.hash(name) still equals name. The whole pack is hashed; the REST + backend computes the hash server-side, so for it nothing is downloaded. The index is hashed first and the packs only if it is intact. The packs could be hashed even with a corrupt index, but a corrupt index already means the user has to repair it, and that @@ -1137,7 +1142,7 @@ def check(self, repair=False, max_duration=0, max_age=0, repo_only=False): far too slow and expensive for a routine (e.g. cron) check. With repair=True and a corrupt index, and if every pack is intact, the index is rebuilt from the packs' object headers and persisted; on a full check the archives phase rebuilds and re-persists it afterwards, see - ArchiveChecker.finish. Packs are verified by sha256, which is content-addressing rather than a + ArchiveChecker.finish. Packs are verified by blake3, which is content-addressing rather than a MAC, so this rebuild detects accidental corruption but not tampering, refs #9901, #10026. If any pack is corrupt the index is left unchanged, refs #8572, #10026. Pack ids found corrupt are kept in cache/checked-packs, refs #9696. @@ -1164,14 +1169,14 @@ def check(self, repair=False, max_duration=0, max_age=0, repo_only=False): """ def verify(namespace, name): - # name is the sha256 of the object's content, so it is intact iff store.hash() matches. + # name is the blake3 hash of the object's content, so it is intact iff store.hash() matches. key = f"{namespace}/{name}" try: - ok = self.store.hash(key) == name + ok = self.store.hash(key, algorithm=NAME_HASH) == name except StoreObjectNotFound: return True # vanished since store.list(); not an error if not ok: - logger.error(f"Store object {key} is corrupted: content does not match its name (sha256).") + logger.error(f"Store object {key} is corrupted: content does not match its name ({NAME_HASH}).") return ok def store_list(namespace): @@ -1330,7 +1335,7 @@ def recorded_ts(info): # build_chunkindex_from_repo matches this verification. write_immediately persists the # index and drops the corrupt fragments. # the walk gets no validator: validating needs the key, which a Repository does not - # have. A pack is named by the sha256 of its content, so a pack damaged in the store + # have. A pack is named by the blake3 hash of its content, so a pack damaged in the store # fails verify() above and pack_errors > 0 keeps it out of here. A pack that matches # its name and still has a bad object header makes iter_headers raise, see #10026. build_chunkindex_from_repo(self, slow_rebuild=True, write_immediately=True) @@ -1628,12 +1633,12 @@ def compact_pack(self, pack_id, *, keep_ids: set, drop_ids: set, chunks=None): if cursor < pack_size: sources.append((pack_hex, cursor, pack_size - cursor)) - # write the new pack (named sha256 of its content) from those spans before touching the index + # write the new pack (named blake3 of its content) from those spans before touching the index # or the old pack, so a failed read-back leaves everything unchanged. a span reading back short # (defrag raises ReadRangeError) means the pack file is truncated or corrupt. if sources: try: - new_pack_id = hex_to_bin(self.store.defrag(sources, algorithm="sha256", namespace="packs")) + new_pack_id = hex_to_bin(self.store.defrag(sources, algorithm=NAME_HASH, namespace="packs")) except ReadRangeError as e: raise IntegrityError(f'pack {pack_hex}: {e}, run "borg check"') from e else: @@ -1733,7 +1738,7 @@ def merge_packs(self, pack_ids, *, chunks=None, max_size=None): if current: batches.append(current) - # write each batch as a new pack (named sha256 of its content) and repoint its objects: an + # write each batch as a new pack (named blake3 of its content) and repoint its objects: an # object's new offset is the running byte total of the packs before its pack in the batch, # plus its old offset within that pack. pi = ProgressIndicatorPercent(total=len(batches), msg="Merging packs %3.0f%%", msgid="repository.merge_packs") @@ -1744,7 +1749,7 @@ def merge_packs(self, pack_ids, *, chunks=None, max_size=None): self._lock_refresh() # refresh the lock per batch, the loop can run for a while sources = [(bin_to_hex(pid), 0, pack_size[pid]) for pid in batch] try: - new_pack_id = hex_to_bin(self.store.defrag(sources, algorithm="sha256", namespace="packs")) + new_pack_id = hex_to_bin(self.store.defrag(sources, algorithm=NAME_HASH, namespace="packs")) except ReadRangeError as e: # a source pack shrank or is corrupt raise IntegrityError(f'merge_packs: {e}, run "borg check"') from e produced.add(new_pack_id) @@ -1790,7 +1795,7 @@ def transform_pack(self, pack_id, ids, transform, *, chunks=None, before_change= IntegrityError, before anything is written. If every object is kept and no gap bytes are dropped, the store and the chunk index are not - touched at all. Otherwise the new pack (named sha256 of its content) is stored, the indexed + touched at all. Otherwise the new pack (named blake3 of its content) is stored, the indexed objects are repointed at it, and the old pack is deleted last, so the objects' bytes are never the only copy. @@ -1852,7 +1857,7 @@ def transform_pack(self, pack_id, ids, transform, *, chunks=None, before_change= if not changed: return pack_id, pack_size pack_data = b"".join(pieces) - new_pack_id = sha256(pack_data).digest() + new_pack_id = blake3_256(pack_data) if new_pack_id == pack_id: # the transforms reproduced the pack byte-identically return pack_id, pack_size diff --git a/src/borg/storelocking.py b/src/borg/storelocking.py index a2a8daccd1..ff323d2f60 100644 --- a/src/borg/storelocking.py +++ b/src/borg/storelocking.py @@ -72,7 +72,6 @@ """ import datetime -import hashlib import json import random import threading @@ -83,6 +82,7 @@ from . import platform from .constants import MAX_MUTUAL_CLOCK_SKEW +from .crypto.key import blake3_256_hex from .helpers import Error, ErrorWithTraceback, format_timedelta from .logger import create_logger @@ -221,7 +221,7 @@ def _create_lock(self, *, exclusive=None, dt=None, update_last_refresh=False): timestamp = now.isoformat(timespec="milliseconds") lock = dict(exclusive=exclusive, hostid=self.id[0], processid=self.id[1], threadid=self.id[2], time=timestamp) value = json.dumps(lock).encode("utf-8") - key = hashlib.sha256(value).hexdigest() + key = blake3_256_hex(value) logger.debug(f"LOCK-CREATE: creating lock in store. key: {key}, lock: {lock}.") self.store.store(f"locks/{key}", value) if update_last_refresh: diff --git a/src/borg/testsuite/archiver/check_cmd_test.py b/src/borg/testsuite/archiver/check_cmd_test.py index d71624f548..339ca0ba16 100644 --- a/src/borg/testsuite/archiver/check_cmd_test.py +++ b/src/borg/testsuite/archiver/check_cmd_test.py @@ -92,8 +92,8 @@ def test_check_soft_interrupt(archivers, request, monkeypatch): orig_hash = repository.store.hash pack_checks = [] - def hash_then_interrupt(key): - result = orig_hash(key) + def hash_then_interrupt(key, **kwargs): + result = orig_hash(key, **kwargs) if key.startswith("packs/"): # count pack checks, not the index files hashed first pack_checks.append(key) if len(pack_checks) == 1: # one Ctrl-C after the first pack is checked @@ -183,8 +183,8 @@ def test_check_interrupt_skips_archive_check(archivers, request, monkeypatch): orig_hash = Store.hash pack_checks = [] - def hash_then_interrupt(self, key): - result = orig_hash(self, key) + def hash_then_interrupt(self, key, **kwargs): + result = orig_hash(self, key, **kwargs) if key.startswith("packs/"): # count pack checks, not the index files hashed first pack_checks.append(key) if len(pack_checks) == 1: # one Ctrl-C after the first pack is checked @@ -623,8 +623,8 @@ def test_check_repair_rebuilds_corrupt_index(archivers, request): with repository: index_infos = list(repository.store_list("index")) assert index_infos # a fresh index was persisted - for info in index_infos: # each fragment's content still matches its sha256 name - assert repository.store.hash(f"index/{info.name}") == info.name + for info in index_infos: # each fragment's content still matches its blake3 name + assert repository.store.hash(f"index/{info.name}", algorithm="blake3") == info.name cmd(archiver, "check", exit_code=0) # the repository is consistent again assert "archive1" in cmd(archiver, "repo-list") # and remains usable @@ -772,7 +772,7 @@ def test_repair_resyncs_pack_with_corrupt_object_header(archivers, request, dama assert (chunk_id in repository.chunks) == (chunk_id != damaged_id) cmd(archiver, "list", "archive1", exit_code=0) # the archives are readable # the pack still holds the damaged bytes, so it keeps failing the store-level check: a pack is - # named by the sha256 of its content. Repairing that is repository-level repair (#10026). + # named by the blake3 hash of its content. Repairing that is repository-level repair (#10026). output = cmd(archiver, "check", "--repository-only", exit_code=1) assert f"Store object packs/{bin_to_hex(pack_id)} is corrupted" in output @@ -892,7 +892,7 @@ def build_chunkindex_from_repo(repository, **kwargs): monkeypatch.setattr(ArchiveChecker, "make_key", make_key) monkeypatch.setattr(archive_module, "build_chunkindex_from_repo", build_chunkindex_from_repo) # --archives-only: the repository check would stop at the damaged pack (a pack is named by the - # sha256 of its content) before the archives check ever walks it. + # blake3 hash of its content) before the archives check ever walks it. with pytest.raises(CorruptPack) as excinfo: cmd(archiver, "check", "--archives-only") assert f"no object header at offset {damaged_offset} (pack corruption)" in str(excinfo.value) @@ -1102,7 +1102,7 @@ def test_corrupted_file_chunk(archivers, request, init_args): @pytest.mark.skip( - reason="TODO: a non-repair check verifies index and packs by sha256 and uses that verified index (it does " + reason="TODO: a non-repair check verifies index and packs by content hash and uses that verified index (it does " "not rebuild it); after dropping all packs the index still lists their chunks, so reading them raises " "ObjectNotFound instead of being reported as missing. Needs the index/repair redesign, refs #8572." ) diff --git a/src/borg/testsuite/archiver/key_cmds_test.py b/src/borg/testsuite/archiver/key_cmds_test.py index c553761cc9..0da33879cc 100644 --- a/src/borg/testsuite/archiver/key_cmds_test.py +++ b/src/borg/testsuite/archiver/key_cmds_test.py @@ -1,6 +1,6 @@ import binascii import os -from hashlib import sha256 +from blake3 import blake3 import pytest @@ -101,17 +101,17 @@ def test_change_location_authenticated_to_repokey(archivers, request): assert "(repokey, authenticated-sha256)" in log -def test_keyfile_name_is_content_sha256(archivers, request): +def test_keyfile_name_is_content_blake3(archivers, request): archiver = request.getfixturevalue(archivers) cmd(archiver, "repo-create", KF_ENCRYPTION, KF_LOCATION) [key_filename] = os.listdir(archiver.keys_path) key_path = os.path.join(archiver.keys_path, key_filename) with open(key_path, "rb") as fd: key_content = fd.read() - assert key_filename == sha256(key_content).hexdigest() + assert key_filename == blake3(key_content).hexdigest() -def test_change_passphrase_renames_keyfile_to_new_sha256(archivers, request): +def test_change_passphrase_renames_keyfile_to_new_blake3(archivers, request): archiver = request.getfixturevalue(archivers) cmd(archiver, "repo-create", KF_ENCRYPTION, KF_LOCATION) [old_key_filename] = os.listdir(archiver.keys_path) @@ -125,7 +125,7 @@ def test_change_passphrase_renames_keyfile_to_new_sha256(archivers, request): assert not os.path.exists(old_key_path) with open(new_key_path, "rb") as fd: key_content = fd.read() - assert new_key_filename == sha256(key_content).hexdigest() + assert new_key_filename == blake3(key_content).hexdigest() cmd(archiver, "repo-list") diff --git a/src/borg/testsuite/cache_test.py b/src/borg/testsuite/cache_test.py index 2e074cd30a..a1095951dc 100644 --- a/src/borg/testsuite/cache_test.py +++ b/src/borg/testsuite/cache_test.py @@ -4,6 +4,7 @@ from datetime import UTC, datetime import pytest +from blake3 import blake3 from .hashindex_test import H from .crypto.key_test import TestKey @@ -152,7 +153,7 @@ def test_read_chunkindex_from_repo_corrupt(tmp_path): repository_location = os.fspath(tmp_path / "repository") with Repository(repository_location, exclusive=True, create=True) as repository: content = b"not a serialized chunk index" - name = hashlib.sha256(content).hexdigest() # valid name, so the name check passes + name = blake3(content).hexdigest() # valid name, so the name check passes repository.store_store(f"index/{name}", content) with pytest.raises(CorruptChunkIndexFragment): read_chunkindex_from_repo(repository, name) @@ -166,7 +167,7 @@ def test_build_chunkindex_rebuilds_on_corrupt_fragment(tmp_path): ci[H(1)] = ChunkIndexEntry(ChunkIndex.F_NEW, 0, H(1), 0, 4) write_chunkindex_to_repo(repository, ci, incremental=False, force_write=True) content = b"not a serialized chunk index" - name = hashlib.sha256(content).hexdigest() + name = blake3(content).hexdigest() repository.store_store(f"index/{name}", content) chunks = build_chunkindex_from_repo(repository) # the rebuild reads the (empty) packs namespace, so H(1) from the corrupt fragment is absent diff --git a/src/borg/testsuite/repository_test.py b/src/borg/testsuite/repository_test.py index 7fa3c4ab9f..7c889ebad3 100644 --- a/src/borg/testsuite/repository_test.py +++ b/src/borg/testsuite/repository_test.py @@ -5,7 +5,7 @@ import sys import time from collections import namedtuple -from hashlib import sha256 +from blake3 import blake3 import pytest from borghash import HashTableNT @@ -88,7 +88,7 @@ def reopen(repository, exclusive: bool | None = True, create=False): def fchunk(data, meta=b"", chunk_id=b"\x00" * 32): # Build a raw chunk with a valid RepoObj layout but no encryption or compression. Pass a unique - # chunk_id when objects must not share a pack: identical bytes hash to the same sha256 pack id + # chunk_id when objects must not share a pack: identical bytes hash to the same blake3 pack id # and would otherwise collapse into one pack. hdr = RepoObj.obj_header.pack(OBJ_MAGIC, OBJ_VERSION, chunk_id, len(meta), len(data)) assert isinstance(data, bytes) @@ -422,7 +422,7 @@ def test_compact_pack_drops_whole_pack(repo_fixtures, request): def test_compact_pack_keep_all_is_noop(repo_fixtures, request): - # Keeping every object reproduces the same pack: same sha256 name, old pack not deleted. Ids passed + # Keeping every object reproduces the same pack: same blake3 name, old pack not deleted. Ids passed # out of order must give the same result, since compact_pack sorts by offset. chunk0 = fchunk(b"DATA0", chunk_id=H(0)) chunk1 = fchunk(b"DATA1", chunk_id=H(1)) @@ -781,7 +781,7 @@ def test_pack_writer_n1_flush(): assert len(results) == 1 stored_id, pack_id, obj_offset, obj_size = results[0] assert stored_id == chunk_id - assert pack_id == sha256(cdata).digest() + assert pack_id == blake3(cdata).digest() assert obj_offset == 0 assert obj_size == len(cdata) @@ -796,7 +796,7 @@ def test_pack_writer_n2_flush(): assert results is not None assert len(results) == 2 pack_data = data1 + data2 - expected_pack_id = sha256(pack_data).digest() + expected_pack_id = blake3(pack_data).digest() assert results[0] == (id1, expected_pack_id, 0, len(data1)) assert results[1] == (id2, expected_pack_id, len(data1), len(data2)) @@ -873,10 +873,10 @@ def test_pack_writer_async_defers_results(): pw = PackWriter(store, max_count=1, chunks=chunks) assert pw.add(id1, data1) is None # pack 1 handed to the store-thread, nothing to report yet results = pw.add(id2, data2) # joins pack 1's store, hands off pack 2 - assert results == [(id1, sha256(data1).digest(), 0, len(data1))] + assert results == [(id1, blake3(data1).digest(), 0, len(data1))] assert not chunks.is_pending(id1) # the join resolved pack 1's entries - assert pw.add(id3, data3) == [(id2, sha256(data2).digest(), 0, len(data2))] - assert pw.flush() == [(id3, sha256(data3).digest(), 0, len(data3))] # barrier: joins pack 3 + assert pw.add(id3, data3) == [(id2, blake3(data2).digest(), 0, len(data2))] + assert pw.flush() == [(id3, blake3(data3).digest(), 0, len(data3))] # barrier: joins pack 3 for chunk_id in (id1, id2, id3): assert not chunks.is_pending(chunk_id) @@ -989,7 +989,7 @@ def test_put_marks_id_in_chunk_index(tmp_path): repository.flush() entry = repository._chunks.get(id1) assert not repository._chunks.is_pending(id1) - assert entry.pack_id == sha256(fchunk(b"ZEROS")).digest() + assert entry.pack_id == blake3(fchunk(b"ZEROS")).digest() assert entry.size == 0 # uncompressed size filled in by cache layer @@ -1031,7 +1031,7 @@ def boom(*args, **kwargs): def _serialized_chunkindex(): - # Serialize an empty ChunkIndex to bytes, as stored under index/. check() parses + # Serialize an empty ChunkIndex to bytes, as stored under index/. check() parses # index fragments, so a fragment must be a real ChunkIndex serialization. with io.BytesIO() as f: ChunkIndex().write(f) @@ -1040,11 +1040,11 @@ def _serialized_chunkindex(): def test_check_detects_corruption_in_later_object(tmp_path): # Corruption anywhere in a multi-object pack must be caught, not just in the first object: the pack - # is named by sha256(content), so flipping any byte makes its stored hash differ from its name. + # is named by blake3(content), so flipping any byte makes its stored hash differ from its name. chunk1 = fchunk(b"FIRST", chunk_id=H(1)) chunk2 = fchunk(b"SECOND", chunk_id=H(2)) pack = chunk1 + chunk2 - pack_name = "packs/" + bin_to_hex(sha256(pack).digest()) + pack_name = "packs/" + blake3(pack).hexdigest() with Repository(str(tmp_path / "repo"), exclusive=True, create=True) as repository: repository.store_store(pack_name, pack) assert repository.check(repair=False) is True # both objects are intact @@ -1057,12 +1057,12 @@ def test_check_detects_corruption_in_later_object(tmp_path): def test_check_detects_index_corruption(tmp_path): - # index/ objects are named by sha256(content) like packs, so check verifies them the same way. + # index/ objects are named by blake3(content) like packs, so check verifies them the same way. content = _serialized_chunkindex() - index_name = "index/" + bin_to_hex(sha256(content).digest()) + index_name = "index/" + blake3(content).hexdigest() with Repository(str(tmp_path / "repo"), exclusive=True, create=True) as repository: repository.store_store(index_name, content) - assert repository.check(repair=False) is True # index object intact (name == sha256(content)) + assert repository.check(repair=False) is True # index object intact (name == blake3(content)) corrupted = bytearray(content) corrupted[0] ^= 0xFF @@ -1096,7 +1096,7 @@ def test_check_repair_rebuilds_corrupt_index(tmp_path): with reopen(repository) as repository: index_names = [f"index/{info.name}" for info in repository.store_list("index")] assert index_names # close() persisted at least one index fragment - for name in index_names: # rot every fragment so its content no longer matches its sha256 name + for name in index_names: # rot every fragment so its content no longer matches its blake3 name data = bytearray(repository.store_load(name)) data[0] ^= 0xFF repository.store_store(name, bytes(data)) @@ -1121,7 +1121,7 @@ def test_check_repair_refuses_when_pack_corrupt(tmp_path): with reopen(repository) as repository: bad_pack_name = "packs/" + bin_to_hex(repository.chunks[H(2)].pack_id) data = bytearray(repository.store_load(bad_pack_name)) - data[-1] ^= 0xFF # rot the pack holding H(2): its content no longer matches its sha256 name + data[-1] ^= 0xFF # rot the pack holding H(2): its content no longer matches its blake3 name repository.store_store(bad_pack_name, bytes(data)) for info in repository.store_list("index"): # rot the index so repair takes the rebuild path name = f"index/{info.name}" @@ -1198,7 +1198,7 @@ def test_check_intact_multi_object_pack_passes(tmp_path): # An intact pack with several objects passes: it is hashed as a whole, so the object count # does not matter. pack = fchunk(b"A", chunk_id=H(1)) + fchunk(b"BB", chunk_id=H(2)) + fchunk(b"CCC", chunk_id=H(3)) - pack_name = "packs/" + bin_to_hex(sha256(pack).digest()) + pack_name = "packs/" + blake3(pack).hexdigest() with Repository(str(tmp_path / "repo"), exclusive=True, create=True) as repository: repository.store_store(pack_name, pack) assert repository.check(repair=False) is True @@ -1247,9 +1247,9 @@ def test_check_reports_orphan_pack_not_referenced_by_index(tmp_path, caplog): repository.put(H(x), fchunk(b"DATA-%02d" % x, chunk_id=H(x))) repository.flush() # flush before close persists the index with Repository(location, exclusive=True) as repository: - # a validly-named pack (name == sha256(content)) that no index entry points into. + # a validly-named pack (name == blake3(content)) that no index entry points into. content = b"orphan pack content" - orphan_id = sha256(content).digest() + orphan_id = blake3(content).digest() repository.store_store("packs/" + bin_to_hex(orphan_id), content) with caplog.at_level(logging.INFO): assert repository.check(repair=False) is True @@ -1269,7 +1269,7 @@ def test_check_missing_pack_detection_skipped_when_index_unreadable(tmp_path, ca pack_id = repository.chunks[H(0)].pack_id repository.store_delete("packs/" + bin_to_hex(pack_id)) # pack gone, index entry kept content = b"not a serialized chunk index" - repository.store_store("index/" + bin_to_hex(sha256(content).digest()), content) + repository.store_store("index/" + blake3(content).hexdigest(), content) with caplog.at_level(logging.WARNING): assert repository.check(repair=False) is True assert "Missing pack" not in caplog.text @@ -1306,7 +1306,7 @@ def test_check_checked_packs_roundtrip(tmp_path): assert tuple(loaded.table[H(2)]) == (456, 0) corrupted = bytearray(repository.store.load(PackTracker.NAME)) - corrupted[0] ^= 0xFF # break the appended sha256 + corrupted[0] ^= 0xFF # break the appended blake3 hash repository.store.store(PackTracker.NAME, bytes(corrupted)) rotted = PackTracker.load(repository.store) assert len(rotted) == 0 @@ -1315,7 +1315,7 @@ def test_check_checked_packs_roundtrip(tmp_path): def test_check_partial_rechecks_pack_sorting_before_checked_one(tmp_path): # a partial check verifies a new pack even when its id sorts before an already-checked pack. intact = fchunk(b"INTACT", chunk_id=H(1)) - intact_id = sha256(intact).digest() + intact_id = blake3(intact).digest() with Repository(str(tmp_path / "repo"), exclusive=True, create=True) as repository: repository.store_store("packs/" + bin_to_hex(intact_id), intact) @@ -1350,18 +1350,18 @@ def _spy_hash(repository, monkeypatch): hashed_keys = [] orig_hash = repository.store.hash - def spy_hash(key): + def spy_hash(key, **kwargs): hashed_keys.append(key) - return orig_hash(key) + return orig_hash(key, **kwargs) monkeypatch.setattr(repository.store, "hash", spy_hash) return hashed_keys def _store_intact_pack(repository, chunk_id=H(1)): - # a distinct chunk_id yields distinct bytes and thus a distinct sha256 pack id. + # a distinct chunk_id yields distinct bytes and thus a distinct blake3 pack id. intact = fchunk(b"INTACT", chunk_id=chunk_id) - intact_id = sha256(intact).digest() + intact_id = blake3(intact).digest() pack_key = "packs/" + bin_to_hex(intact_id) repository.store_store(pack_key, intact) return intact_id, pack_key @@ -1514,9 +1514,9 @@ def test_check_partial_break_reports_unreached_corrupt_record(tmp_path, monkeypa hashed_keys = [] orig_hash = repository.store.hash - def hash_and_advance(key): + def hash_and_advance(key, **kwargs): hashed_keys.append(key) - result = orig_hash(key) + result = orig_hash(key, **kwargs) if key == intact_key: clock["t"] = 10**9 return result @@ -1669,9 +1669,9 @@ def test_check_max_age_prunes_vanished_ok_record(tmp_path): def test_check_max_age_partial_progress(tmp_path, monkeypatch): # a partial check with max_age skips packs with a fresh intact record and verifies the rest. pack_a = fchunk(b"A", chunk_id=H(1)) - pack_a_id = sha256(pack_a).digest() + pack_a_id = blake3(pack_a).digest() pack_b = fchunk(b"BB", chunk_id=H(2)) - pack_b_id = sha256(pack_b).digest() + pack_b_id = blake3(pack_b).digest() with Repository(str(tmp_path / "repo"), exclusive=True, create=True) as repository: repository.store_store("packs/" + bin_to_hex(pack_a_id), pack_a) repository.store_store("packs/" + bin_to_hex(pack_b_id), pack_b) @@ -1734,7 +1734,7 @@ def test_check_max_age_reuses_records_of_plain_check(tmp_path, monkeypatch): def test_check_checked_packs_ignores_foreign_entry_layout(tmp_path): - # load() drops a set whose entries have a different layout than Entry, even though its sha256 matches. + # load() drops a set whose entries have a different layout than Entry, even though its blake3 hash matches. OtherEntry = namedtuple("OtherEntry", "timestamp result extra") OtherFormat = namedtuple("OtherFormat", "timestamp result extra") with Repository(str(tmp_path / "repo"), exclusive=True, create=True) as repository: @@ -1745,7 +1745,7 @@ def test_check_checked_packs_ignores_foreign_entry_layout(tmp_path): with io.BytesIO() as f: table.write(f) data = f.getvalue() - repository.store_store(PackTracker.NAME, data + sha256(data).digest()) + repository.store_store(PackTracker.NAME, data + blake3(data).digest()) tracker = PackTracker.load(repository.store) assert len(tracker) == 0 @@ -1771,9 +1771,9 @@ def finish(self, *args, **kwargs): monkeypatch.setattr("borg.repository.ProgressIndicatorPercent", FakePI) pack = fchunk(b"A", chunk_id=H(1)) - pack_name = "packs/" + bin_to_hex(sha256(pack).digest()) + pack_name = "packs/" + blake3(pack).hexdigest() index_content = _serialized_chunkindex() - index_name = "index/" + bin_to_hex(sha256(index_content).digest()) + index_name = "index/" + blake3(index_content).hexdigest() with Repository(str(tmp_path / "repo"), exclusive=True, create=True) as repository: repository.store_store(pack_name, pack) repository.store_store(index_name, index_content) @@ -1788,8 +1788,8 @@ def finish(self, *args, **kwargs): assert pi.position == pi.total -def test_pack_writer_final_partial_pack_uses_sha256(): - # A final flush with fewer pieces than max_count must still use SHA256(pack_bytes). +def test_pack_writer_final_partial_pack_uses_blake3(): + # A final flush with fewer pieces than max_count must still use blake3(pack_bytes). store = MockStore() chunk_id = b"d" * 32 cdata = b"solo" @@ -1799,7 +1799,7 @@ def test_pack_writer_final_partial_pack_uses_sha256(): assert results is not None assert len(results) == 1 _, pack_id, _, _ = results[0] - assert pack_id == sha256(cdata).digest() + assert pack_id == blake3(cdata).digest() assert pack_id != chunk_id