refactor(aggregator): deterministic snapshots archives

[Alenar]

Content

This PR harden the archives building of the mithril-aggregator in order to makes them bytes identical for a given content.

It extract the FileArchiver api from the Aggregator into introduces a new library, mithril-file-archiver, with additional changes to ensure deterministic, byte-stable .tar.zst archives creations.

Main Changes

New Library Added:

• Introduces mithril-file-archiver as a standalone library for generating deterministic archives. Like the previous api, it supports archiving files, directories, or serializable data with compression.

  • Includes support for configurable Zstandard compression parameters.
  • Ensures reproducibility and determinism across platforms.
  • Provides test utilities, such as golden master tests and reproducibility checks.- Tests Added/Updated:
  • Comprehensive tests for functionality, compatibility, cross-platform reproducibility, and golden master validation were implemented in the new library.

• Existing File Archiver Logic Removed: The previously internal file_archiver and associated modules (appender, tools, etc.) in the mithril-aggregator folder have been moved to the new library.

  • Updated mithril-aggregator to integrate mithril-file-archiver for tasks like snapshot generation and ancillary file handling.
  • Adjusted tests to use the new library's provided abstractions and utilities (e.g., FileArchiver::new_with_default_parameters and FileArchiveTestExtension). - Existing tests in mithril-aggregator were adapted accordingly.
  • Removed direct dependencies on zstd and tar in mithril-aggregator.

• Documentation:

  • Enhanced documentation to describe mithril-file-archiver, its API, and usage constraints.

• Other:

  • follow_symlinks support is now disabled in the FileArchiver (as recommended in their doc)
  • some unused dependencies were removed from the aggregator: tempfile, and mithril-test-http-server

How deterministic archives were achieved

• Changes to the tar header:
  • switch to HeaderMode::Deterministic: this set the mtime to a fix value (timestamp 1153704088, date: Jul 23, 2006), the guid and gid to 0, and the permissions to 644 for files and 755 for directories (see the code here)
  • sparse file support is disabled as it's not consistent across systems and the zstandard will compress the sparse part itself anyway
• A normalization path is now run on the AppenderEntries entries list:
  • removal of trailing / from directory names (or \ on Windows)
  • removal of leading ./ from path
  • the entries list is now sorted using rust default sort
• changes to AppenderData metadata:
  • a deterministic mtime is set instead of the current time: randomly choose to 1767225600 (2026-01-01 00:00:00 UTC)
• Removal of AppenderDirAll: the order of the files and folder is not stable across systems, if needed, it could be re-created as a constructor of AppenderEntry (from_directory) which would do the same job of recursively adding all entries but would then leverage its normalization pass

Pre-submit checklist

• Branch
  • Tests are provided (if possible)
  • Crates versions are updated (if relevant)
  • CHANGELOG file is updated (if relevant)
  • Commit sequence broadly makes sense
  • Key commits have useful messages
• PR
  • All check jobs of the CI have succeeded
  • Self-reviewed the diff
  • Useful pull request description
  • Reviewer requested
• Documentation
  • Update README file (if relevant)
  • No new TODOs introduced

Issue(s)

Relates to #3435

[github-actions]

Test Results

     5 files  ± 0     221 suites  +12   58m 6s ⏱️ - 1h 28m 25s
 3 400 tests  - 50   3 400 ✅  - 50  0 💤 ±0  0 ❌ ±0 
11 354 runs  +41  11 354 ✅ +41  0 💤 ±0  0 ❌ ±0 

Results for commit c7ed6f2. ± Comparison against base commit 58b575a.

This pull request removes 109 and adds 59 tests. Note that renamed tests count towards both.

mithril-aggregator ‑ tools::file_archiver::api::tests::compute_size_of_uncompressed_data_and_archive
mithril-aggregator ‑ tools::file_archiver::api::tests::overwrite_already_existing_archive_when_archiving_succeed
mithril-aggregator ‑ tools::file_archiver::api::tests::should_create_a_valid_archive_with_zstandard_compression
mithril-aggregator ‑ tools::file_archiver::api::tests::should_delete_tmp_file_in_target_directory_if_archiving_fail
mithril-aggregator ‑ tools::file_archiver::api::tests::should_not_delete_an_already_existing_archive_with_same_name_if_archiving_fail
mithril-aggregator ‑ tools::file_archiver::appender::tests::appender_data::append_serializable_json
mithril-aggregator ‑ tools::file_archiver::appender::tests::appender_data::appended_entry_have_read_write_permissions_and_time_metadata
mithril-aggregator ‑ tools::file_archiver::appender::tests::appender_data::compute_uncompressed_size
mithril-aggregator ‑ tools::file_archiver::appender::tests::appender_dir_all::compute_uncompressed_size
mithril-aggregator ‑ tools::file_archiver::appender::tests::appender_entries::can_append_duplicate_files_and_directories
…

mithril-file-archiver ‑ api::tests::compute_size_of_uncompressed_data_and_archive
mithril-file-archiver ‑ api::tests::overwrite_already_existing_archive_when_archiving_succeed
mithril-file-archiver ‑ api::tests::should_create_a_valid_archive_with_zstandard_compression
mithril-file-archiver ‑ api::tests::should_delete_tmp_file_in_target_directory_if_archiving_fail
mithril-file-archiver ‑ api::tests::should_not_delete_an_already_existing_archive_with_same_name_if_archiving_fail
mithril-file-archiver ‑ appender::tests::appender_data::append_serializable_json
mithril-file-archiver ‑ appender::tests::appender_data::appended_entry_have_read_write_permissions_and_fixed_time_metadata
mithril-file-archiver ‑ appender::tests::appender_data::compute_uncompressed_size
mithril-file-archiver ‑ appender::tests::appender_entries::can_append_duplicate_files_and_directories
mithril-file-archiver ‑ appender::tests::appender_entries::compute_uncompressed_size_of_its_paths
…

♻️ This comment has been updated with latest results.