repository: single config/config text object (repo version 5)

docs/faq.rst

@@ -30,7 +30,7 @@ Can I back up from multiple servers into a single repository?
 Yes, you can! Even simultaneously.
 
 The clocks of machines sharing a repository should be roughly synchronized
-(e.g. via NTP): repository locks and archive/manifest timestamps are based on
+(e.g. via NTP): repository locks and archive timestamps are based on
 the clients' clocks, so big clock differences between clients can cause
 trouble. Where the storage backend provides object timestamps (file, sftp, s3
 and current rest servers - but not rclone), borg cross-checks lock staleness

docs/internals.rst

@@ -9,8 +9,8 @@ of Borg.
 
 Borg uses a low-level, key-value store, the :ref:`repository`, and
 implements a more complex data structure on top of it, which is made
-up of the :ref:`manifest <manifest>`, :ref:`archives <archive>`,
-:ref:`items <item>` and data :ref:`chunks`.
+up of the :ref:`archives <archive>`, :ref:`items <item>` and data
+:ref:`chunks`.
 
 Each repository can hold multiple :ref:`archives <archive>`, which
 represent individual backups that contain a full archive of the files

docs/internals/data-structures.rst

@@ -38,14 +38,8 @@ It is the same for every repository and independent of the key/encryption mode
 (unlike the chunk id hash, which the key mode selects).
 
 config/
-  readme
-    simple text object telling that this is a Borg repository
-  id
-    the unique repository ID encoded as hexadecimal number text
-  version
-    the repository version encoded as decimal number text
-  manifest
-    the manifest (see :ref:`manifest`), a repository object, binary
+  config
+    the repository config (see :ref:`repo_config`), a text object
   space-reserve.N
     purely random binary data to reserve space, e.g. for disk-full emergencies.
     These objects are created and removed by ``borg repo-space``.
@@ -177,10 +171,9 @@ Repo object metadata
 
 Metadata is a MessagePack-encoded (and encrypted/authenticated) dict with:
 
-- type (the repo object type, a one-character string: ``M`` manifest,
-  ``A`` archive metadata, ``C`` archive metadata stream chunk ids,
-  ``S`` archive metadata stream chunk, ``F`` file content stream chunk -
-  see the ``ROBJ_*`` constants)
+- type (the repo object type, a one-character string: ``A`` archive metadata,
+  ``C`` archive metadata stream chunk ids, ``S`` archive metadata stream chunk,
+  ``F`` file content stream chunk - see the ``ROBJ_*`` constants)
 - ctype (compression type 0..255)
 - clevel (compression level, one byte, interpreted depending on ctype - see
   :ref:`data-compression`)
@@ -235,47 +228,64 @@ More on how this helps security in :ref:`security_structural_auth`.
     :figwidth: 100%
     :width: 100%
 
-.. _manifest:
+.. _repo_config:
 
-The manifest
-~~~~~~~~~~~~
-
-The manifest is a repository object stored as the ``config/manifest`` store
-object (see Repository_), so it is not inside a pack file and not in the chunks
-index. Different from all other repository objects, the chunk id in its object
-header is not the hash of its content, but all-zero
-(``Manifest.MANIFEST_ID``).
-
-The manifest is written when the repository is created and by ``borg check
---repair`` when it rebuilds a lost or corrupted manifest. Commands that modify
-the repository also call ``Manifest.write()``, but that only stores a new
-manifest object if the content changed. It looks like this:
-
-.. code-block:: python
-
-    {
-        'version': 2,
-        'archives': {},
-        'config': {},
-    }
-
-Borg 2 always writes *version* 2. Reading also accepts version 1, which is what
-borg 1.x repositories have (they are supported read-only, e.g. for
-``borg transfer``).
-
-A *timestamp* entry, as written by borg 1.x and by older borg 2 versions, is
-accepted and ignored when reading.
-
-The *archives* dict is always empty: the list of archives is not part of the
-manifest, each archive has its own pointer object in the ``archives/``
-namespace, see :ref:`archive`.
-
-*config* is a general-purpose location for additional metadata. All versions
-of Borg preserve its contents. Currently, borg does not store anything in there.
+Repository config
+~~~~~~~~~~~~~~~~~
 
-A *config['item_keys']* list (written by older borg 2 versions) or a top-level
-*item_keys* list (borg 1.x) is accepted and ignored when reading: *borg check*
-does not validate item keys against such a list anymore, see Item_.
+The repository config is the ``config/config`` store object (see Repository_), a
+plain text ``INI``-style file. It is the only object borg needs to read to open a
+repository. It looks like this::
+
+    # This is a Borg Backup repository.
+    # See https://borgbackup.readthedocs.io/
+
+    [repository]
+    version = 5
+    id = 0a2744f216526be75ae14a5fa5b123127bb218558219f6203e09e4f220e45903
+    encryption = aes256-ocb
+    id_hash = sha256
+
+*version* is the repository version. borg refuses to open a repository whose
+version it does not support (currently, only version 5 is supported).
+
+*id* is the unique repository ID (32 bytes, hex encoded). It does not change if
+the repository is moved to another location. The keys of the repository are
+bound to it (see :ref:`key_files`), and the client's cache and security
+directories are named after it.
+
+*encryption* and *id_hash* record the crypto suite of the repository's key, by
+the same names ``borg repo-create --encryption`` and ``--id-hash`` accept. This
+is how borg selects the key class when opening a repository, without reading any
+repository object. The config is plaintext and not authenticated; see
+:ref:`remote_access_security` for what protects against a swapped crypto suite.
+Where the key is stored (keyfile or repokey) is not recorded
+here: that is a property of each individual key, see :ref:`key_files`. Both
+entries are present, or none: a repository created via the Python API
+(``Repository.create()``) without a key has none, it can be used as a key/value
+store, but borg refuses to load a key for it.
+
+``borg repo-create`` writes the config once, after the key was created: writing
+it is what makes the store a repository. A store without it (e.g. the leftover of
+an interrupted ``borg repo-create``) is not a repository: borg reports it as not
+a valid repository, and ``borg repo-create`` refuses to create a repository in a
+non-empty location, saying whether it found a repository config there.
+``borg repo-delete --force`` destroys such a store, provided it looks like the
+leftover of an interrupted ``borg repo-create`` (a chunk index, but no packs and
+no archives), so that a leftover can be removed without other access to the
+storage.
+
+The config is the one object ``borg check --repair`` can not restore. If a
+repository lost or damaged its config, it can be recreated by hand: the version
+is 5, the id is in the key (the ``BORG_KEY <id>`` header line of a keyfile or of
+a ``keys/`` object, see :ref:`key_files`), and the encryption mode and id hash
+are what ``borg repo-create`` was given (the key type byte of any repository
+object encodes them as well, see ``KeyType`` in ``constants.py``).
+
+There is no manifest object anymore (borg 1.x had one, holding the archives list
+and identifying the key type): the archives are in the ``archives/`` namespace,
+see :ref:`archive`. borg 1.x repositories are still read (e.g. by
+``borg transfer --from-borg1``) via their manifest.
 
 .. _archive:
 
@@ -1020,8 +1030,7 @@ version
   currently always an integer, 2
 
 repository_id
-  the repository ID, as stored in the repository's ``config/id`` object,
-  see Repository_.
+  the repository ID, as stored in the repository config (see :ref:`repo_config`).
 
 crypt_key
   the initial key material used for the AEAD crypto (512 bits)
@@ -1270,32 +1279,19 @@ the file's name (see :ref:`the files cache <cache>` about that name):
     [cache]
     version = 1
     repository = 3c4...e59
-    manifest = 10e...21c
 
     [integrity]
-    manifest = 10e...21c
     files.9f8...a08 = {"algorithm": "SHA256", "digests": {"final": "e2a...b24"}}
 
 The chunks index is not in this list: it is not a local file, but lives in the
 repository below ``index/`` and has its own integrity mechanism, see
 :ref:`pack-index-namespace`.
 
-The manifest ID is duplicated in the integrity section due to the way all Borg
-versions handle the config file. Instead of creating a "new" config file from
-an internal representation containing only the data understood by Borg,
-the config file is read in entirety (using the Python ConfigParser) and modified.
-This preserves all sections and values not understood by the Borg version
-modifying it.
-
-Thus, if an older versions uses a cache with integrity data, it would preserve
-the integrity section and its contents. If a integrity-aware Borg version
-would read this cache, it would incorrectly report checksum errors, since
-the older version did not update the checksums.
-
-However, by duplicating the manifest ID in the integrity section, it is
-easy to tell whether the checksums concern the current state of the cache.
-If they do not match, borg logs a warning and just does not use the integrity
-data.
+The cache config file is read in its entirety (using the Python ConfigParser),
+modified and written back, so sections and values a Borg version does not
+understand are preserved. There is no guard against an older Borg version
+updating the files cache without updating its integrity data: every Borg
+version that can open a version 5 repository knows the ``[integrity]`` section.
 
 A files cache that fails its integrity check (or can not be read at all) is
 discarded, not used: borg then rebuilds the files cache from the most recent

docs/internals/frontends.rst

@@ -831,6 +831,8 @@ Errors
 
     Repository.AlreadyExists rc: 10 traceback: no
         A repository already exists at {}.
+    Repository.IncompleteRepository rc: 11 traceback: no
+        {} has no repository config: not a borg 2 repository, or the leftover of an interrupted repo-create.
     Repository.CheckNeeded rc: 12 traceback: yes
         Inconsistency detected. Please run "borg check {}".
     Repository.DoesNotExist rc: 13 traceback: no
@@ -900,6 +902,8 @@ Errors
         Passphrase supplied in BORG_PASSPHRASE, by BORG_PASSCOMMAND, or via BORG_PASSPHRASE_FD is incorrect.
     PasswordRetriesExceeded rc: 53 traceback: no
         Exceeded the maximum password retries.
+    RepositoryKeyInfoMissing rc: 54 traceback: no
+        Repository {} has no key information in its config.
 
     CacheInitAbortedError rc: 60 traceback: no
         Cache initialization aborted

docs/internals/packs.rst

@@ -125,7 +125,7 @@ rebuilt the index from it. Rewriting such a pack is repository-level repair, see
 
 ``OBJ_MAGIC`` occurs inside the payloads as well, so the scan accepts a candidate
 only when it validates like any walked header. Validating needs the key, so a
-repair that cannot read the manifest walks without it.
+repair that cannot load the key walks without it.
 
 In the ``none-*`` modes the tag is an unkeyed checksum, and in the
 ``authenticated-*`` modes it binds a blob to its chunk id and nothing else (see
@@ -354,22 +354,19 @@ without decrypting any blob and without the repository key.
 Repository Version
 ------------------
 
-Repositories using pack files require repository version **4**, and the version is the
-only gate for the pack format.
+Repositories using pack files require repository version **4** or later, and the version
+is the only gate for the pack format.
 
-``Repository.create()`` stores ``4`` as the ``config/version`` store object.
+``Repository.save_config()`` stores the version in the repository config (see
+:ref:`repo_config`; currently ``5``, which also introduced the config object itself).
 ``Repository.open()`` reads it back and, if it is not in
-``Repository.acceptable_repo_versions`` (currently ``(4,)``), closes the store again
+``Repository.acceptable_repo_versions`` (currently ``(5,)``), closes the store again
 and raises ``InvalidRepositoryConfig`` -- before any repository data is read. A borg
 version that only accepts version 3 rejects a version 4 repository the same way, so
 the version bump alone locks out every client that does not know about packs.
 
-Borg does have a feature flag mechanism for locking out clients more selectively
-(``Manifest.check_repository_compatibility()``, fed from a ``feature_flags`` entry in
-the manifest ``config`` -- see :ref:`manifest`), but it currently defines no flags at
-all: ``Manifest.SUPPORTED_REPO_FEATURES`` is the empty set, and no borg code writes a
-``feature_flags`` entry. On a repository borg creates, the compatibility check is
-therefore a no-op; there is in particular no ``pack_files`` feature flag.
-
-There is no migration path from version 3 repositories to version 4. Users of the
-version 3 beta format must create a new repository with ``borg repo-create``.
+There is no migration path between the beta repository versions: neither from
+version 3 to 4 (the pack format) nor from version 4 to 5 (the repository config
+object). Users of an older beta format must create a new repository with
+``borg repo-create`` (and can use ``borg transfer`` with a borg version that
+still reads the old repository to copy their archives over).

docs/internals/security.rst

@@ -71,7 +71,7 @@ Above used to be all for borg 1.x and was the reason why it needed the
 tertiary authentication mechanism (TAM) for manifest and archives.
 
 borg 2 now stores the ro_type ("meaning") of a repo object's data into that
-object's metadata (like e.g.: manifest vs. archive vs. user file content data).
+object's metadata (like e.g.: archive metadata vs. user file content data).
 When loading data from the repo, borg verifies that the type of object it got
 matches the type it wanted. borg 2 does not use TAMs any more.
 
@@ -87,7 +87,7 @@ carry an unkeyed checksum rather than a MAC, and an attacker who modifies an
 object can simply recompute it. What still constrains an attacker there is the
 object ID being the (unkeyed) hash of the plaintext: the content of an existing
 object can not be replaced without the ID no longer matching. But the object's
-metadata, the archives list and the manifest are not anchored to anything secret,
+metadata and the archives list are not anchored to anything secret,
 so a ``none-*`` repository provides no tamper protection - only detection of
 accidental corruption.
 
@@ -384,8 +384,29 @@ used:
   repository objects, so the pointer object itself only reveals the archive id (a MAC
   over the archive metadata) plus whatever the store records about it, e.g. its
   modification time.
-- ``config/manifest`` (an encrypted repository object), plus the plaintext
-  ``config/version``, ``config/id`` and ``config/readme``.
+- ``config/config`` -- the plaintext repository config: version, id and the names of
+  the crypto suite (encryption mode, id hash), see :ref:`repo_config`. It is neither
+  encrypted nor authenticated, so an attacker with repository access can rewrite the
+  crypto suite - just like they could replace the manifest object that used to identify
+  the key type in borg 1.x and earlier borg 2 versions, whose type byte was also read
+  before anything was authenticated. What protects against a swapped crypto suite is
+  not this object, but:
+
+  - a swap to a suite that does not encrypt (``none-*``, but also
+    ``authenticated-*``: its key blob carries the same key material and no suite
+    name, so it loads fine) would make the client write plaintext. That is caught
+    by the client's security directory: it records the key type of every
+    repository the client accessed, and borg refuses to continue with
+    ``EncryptionMethodMismatch`` if the suite changed. A repository that does not
+    encrypt and is unknown to the client is only accessed after an explicit
+    confirmation (``BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK``). Note that this
+    requires the client environment to be persistent, see `Attack model`_.
+  - a swap between encrypting suites can not expose plaintext: the client
+    would write new objects with the same secret key material under the other
+    cipher, and merely fail to read the existing objects. On a client that knows
+    the repository, the security directory catches this swap as well. The key blobs
+    themselves are bound to the repository id and unlocked by the passphrase, so a
+    client never ends up using key material of the attacker's choice.
 - ``keys/<store hash>`` -- in ``repokey`` mode, the borg key(s), encrypted with the
   passphrase-derived KEK (see :ref:`key_encryption`).
 - ``locks/*`` and ``cache/*``. Note that the per-archive reference caches
@@ -492,7 +513,7 @@ Note that the msgpack unpackers of the RPC data channel (``get_limited_unpacker(
 kinds ``client`` and ``server``) are deliberately configured with the maximum buffer
 size, because whole repository objects are transferred through them. They therefore
 do not bound the memory a peer can make the other side allocate; the stricter limits
-of that helper apply to manifest, archive and key data.
+of that helper apply to archive and key data.
 
 The msgpack implementation used (msgpack-python) has a good security track record,
 a large test suite and no issues found by fuzzing. It is based on the msgpack-c implementation,

docs/usage/debug.rst

@@ -12,7 +12,6 @@ what their name suggests: put objects into the repository / delete objects from
 Please note:
 
 - they will not update the chunks index about the object
-- they will not update the manifest (so no automatic chunks index resync is triggered)
 - they will not check whether the object is in use (e.g. before delete-obj)
 - they will not update any metadata which may point to the object
 

docs/usage/repo-info.rst

@@ -7,7 +7,7 @@ Examples
     $ borg repo-info
     Repository ID: 0a2744f216526be75ae14a5fa5b123127bb218558219f6203e09e4f220e45903
     Location: /path/to/repo
-    Repository version: 4
+    Repository version: 5
     Encrypted: Yes (repokey, aes256-ocb, sha256)
     Security directory: /home/user/.local/share/borg/security/0a2744f216526be75ae14a5fa5b123127bb218558219f6203e09e4f220e45903
     Cache: /home/user/.cache/borg/0a2744f216526be75ae14a5fa5b123127bb218558219f6203e09e4f220e45903