diff --git a/CHANGELOG.md b/CHANGELOG.md index f557675b2..2070512ff 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +- **R-15 Gate 4D:** the entry gate of the exclusive conversion driver (part a). `begin_conversion` takes the + installation's exclusive admission by mutable borrow, derives the root layout from the scope that admission + was acquired for, re-reads the committed binding, the registry-routed journal key and the root-named + manifest, and returns a `ConversionSession` only for the owner of the lease while the operation is in `ADMIT` + or `CONVERT` with the final inventory captured. It reads no clock, holds no key and writes nothing; moving + the journal under the gate is the next part. PR #1013. - **R-15 Gate 4D:** stage a streamed inventory without handling a journal key (stage two, part b, of the streaming capture). `begin_streamed_capture` routes the journal key from the authenticated root and registry, reads the binding and the committed manifest from the root and returns a `StagingSession` that diff --git a/crates/worldscript-secure-storage/src/authority.rs b/crates/worldscript-secure-storage/src/authority.rs index 66b6bb99a..c411defe8 100644 --- a/crates/worldscript-secure-storage/src/authority.rs +++ b/crates/worldscript-secure-storage/src/authority.rs @@ -555,12 +555,7 @@ pub fn begin_streamed_capture( layout: RootLayout<'_>, begin: SessionBegin<'_>, ) -> Result { - // The root alone names the binding: the catalog pages are not read, so starting a capture of a - // very large inventory does not first materialise a very large catalog. - let live = load_committed_root(fs, provider, layout) - .map_err(AuthorityError::Root)? - .and_then(|view| view.root.live_migration) - .ok_or(AuthorityError::NoLiveMigration)?; + let live = committed_live_migration(fs, provider, layout)?; let key = route_journal_key(fs, provider, layout, begin.journal)?; let capture = { let mut ctx = journal_context(&mut *fs, begin.journal, &key, CandidateConflict::Refuse); @@ -582,6 +577,50 @@ pub fn begin_streamed_capture( }) } +/// The binding the committed root names, or [`AuthorityError::NoLiveMigration`]. +/// +/// The root alone names the binding: the catalog pages are not read, so starting a capture of a very +/// large inventory does not first materialise a very large catalog. +fn committed_live_migration( + fs: &mut F, + provider: &P, + layout: RootLayout<'_>, +) -> Result { + load_committed_root(fs, provider, layout) + .map_err(AuthorityError::Root)? + .and_then(|view| view.root.live_migration) + .ok_or(AuthorityError::NoLiveMigration) +} + +/// The journal state the committed root vouches for: its binding and the manifest it names. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct CommittedJournal { + pub(crate) live: LiveMigration, + pub(crate) manifest: JournalManifest, +} + +/// Reads the committed journal state without a key in the caller's hands. +/// +/// The binding comes from the committed root, the journal key is resolved through the key-epoch +/// registry ([`resolve_journal_key`]) and the manifest is loaded by the exact root-named path, so the +/// binding's digest authenticates the bytes before they are trusted. Nothing is written and no lock is +/// taken: the result is a snapshot, and every later mutation repeats its authority checks under the +/// root lock. +pub(crate) fn read_committed_journal( + fs: &mut F, + provider: &P, + layout: RootLayout<'_>, + journal: JournalSource<'_>, +) -> Result { + let live = committed_live_migration(fs, provider, layout)?; + let key = route_journal_key(fs, provider, layout, journal)?; + let manifest = { + let mut ctx = journal_context(&mut *fs, journal, &key, CandidateConflict::Refuse); + load_authoritative_manifest(&mut ctx, &live).map_err(AuthorityError::Journal)? + }; + Ok(CommittedJournal { live, manifest }) +} + impl StagingSession { /// Stages the next page under the session's key and directory; see [`StreamedCapture::push_page`]. pub fn push_page( diff --git a/crates/worldscript-secure-storage/src/conversion.rs b/crates/worldscript-secure-storage/src/conversion.rs new file mode 100644 index 000000000..86fea7101 --- /dev/null +++ b/crates/worldscript-secure-storage/src/conversion.rs @@ -0,0 +1,155 @@ +//! Gate 4D slice C2a: the entry gate of the exclusive conversion driver (§10.3 `ADMIT`, `CONVERT`). +//! +//! Conversion rewrites every record of the final inventory, so it must run with no ordinary operation +//! in flight. [`begin_conversion`] makes that structural: it takes the installation's +//! [`ExclusiveAdmissionGuard`] by mutable borrow, derives the root layout from the scope that guard +//! was acquired for (the root cannot differ from the one admitted), and returns a +//! [`ConversionSession`] that keeps the borrow, so one admission carries one session and the +//! admission outlives it. A shared guard cannot be passed: +//! +//! ```compile_fail +//! use worldscript_secure_storage::{ +//! begin_conversion, ConversionBegin, KeyProvider, SharedAdmissionGuard, StdFs, +//! }; +//! +//! fn shared_cannot_convert( +//! admission: &mut SharedAdmissionGuard, +//! provider: &P, +//! begin: ConversionBegin<'_>, +//! ) { +//! let _ = begin_conversion(admission, &mut StdFs, provider, begin); +//! } +//! ``` +//! +//! The gate re-reads what the committed root vouches for instead of trusting a caller's copy: the +//! binding comes from the root, the journal key is resolved through the key-epoch registry, and the +//! manifest is loaded by the exact root-named path, so the binding's digest authenticates its bytes +//! ([`read_committed_journal`]). It then refuses unless the operation is in `ADMIT` (conversion can be +//! entered) or `CONVERT` (a resumed conversion), the final inventory was captured, and the committed +//! lease belongs to the caller. A lease owned by another owner is not taken over here: a restart is +//! the existing takeover followed by a fresh begin. No clock is read, so a lease past its expiry is +//! still the caller's while nobody has taken over (the fence arbitrates). +//! +//! This slice writes nothing and holds no key; the steps that move the journal are the next slice. + +use crate::admission::{AdmissionScope, ExclusiveAdmissionGuard}; +use crate::authority::{read_committed_journal, AuthorityError, CommittedJournal, JournalSource}; +use crate::durable::DurableFs; +use crate::journal::{phase_code, JournalManifest, MigrationFence}; +use crate::provider::KeyProvider; +use crate::root::LiveMigration; +use crate::root_store::RootLayout; + +/// What the gate needs from its caller: the scope the admission was acquired for, where the journal +/// lives, and the identity the committed lease must carry. +#[derive(Clone, Copy)] +pub struct ConversionBegin<'a> { + pub scope: AdmissionScope<'a>, + pub journal: JournalSource<'a>, + pub owner_id: &'a str, +} + +/// Why the gate refused. Every refusal is returned before anything is written. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum ConversionError { + /// The guard does not hold the scope: other directories, or directories replaced since acquisition. + NotAdmitted, + /// Reading the committed journal failed (no bound migration, key route, authentication, I/O). + Authority(AuthorityError), + /// The operation finished (`DONE`). + TerminalPhase, + /// The operation is in durable terminal refusal (`RECOVERY_REQUIRED`). + RecoveryRequired, + /// Conversion runs in `ADMIT` and `CONVERT` only. + PhaseNotConvertible, + /// The final inventory was not captured in `ADMIT` (§10.3). + FinalInventoryNotCaptured, + /// The committed lease is owned by another owner, or by none. + ForeignLeaseOwner, +} + +impl From for ConversionError { + fn from(error: AuthorityError) -> Self { + Self::Authority(error) + } +} + +/// The exclusive conversion entry: the committed journal state, read under an admission that no +/// ordinary operation can share. +#[derive(Debug)] +pub struct ConversionSession<'a> { + held: &'a mut ExclusiveAdmissionGuard, + scope: AdmissionScope<'a>, + journal: CommittedJournal, +} + +/// Opens the conversion gate over the committed journal; see the module documentation. +pub fn begin_conversion<'a, F: DurableFs, P: KeyProvider>( + held: &'a mut ExclusiveAdmissionGuard, + fs: &mut F, + provider: &P, + begin: ConversionBegin<'a>, +) -> Result, ConversionError> { + ensure_admitted(held, begin.scope)?; + let layout = RootLayout { + root_dir: begin.scope.root_dir, + }; + let journal = read_committed_journal(fs, provider, layout, begin.journal)?; + // The directories must still be the ones admitted after the reads, not only before them. + ensure_admitted(held, begin.scope)?; + check_convertible(&journal.manifest, begin.owner_id)?; + Ok(ConversionSession { + held, + scope: begin.scope, + journal, + }) +} + +fn ensure_admitted( + held: &ExclusiveAdmissionGuard, + scope: AdmissionScope<'_>, +) -> Result<(), ConversionError> { + if held.guards(scope) { + Ok(()) + } else { + Err(ConversionError::NotAdmitted) + } +} + +fn check_convertible(manifest: &JournalManifest, owner_id: &str) -> Result<(), ConversionError> { + match manifest.phase { + phase_code::DONE => return Err(ConversionError::TerminalPhase), + phase_code::RECOVERY_REQUIRED => return Err(ConversionError::RecoveryRequired), + phase_code::ADMIT | phase_code::CONVERT => {} + _ => return Err(ConversionError::PhaseNotConvertible), + } + if !manifest.final_inventory_captured { + return Err(ConversionError::FinalInventoryNotCaptured); + } + if manifest.lease_owner_id.as_deref() != Some(owner_id) { + return Err(ConversionError::ForeignLeaseOwner); + } + Ok(()) +} + +impl ConversionSession<'_> { + /// The manifest the committed root names. + pub fn manifest(&self) -> &JournalManifest { + &self.journal.manifest + } + + /// The binding the committed root holds for it. + pub fn live(&self) -> &LiveMigration { + &self.journal.live + } + + /// The owner's token at the committed revision. + pub fn fence(&self) -> MigrationFence { + MigrationFence::from_manifest(&self.journal.manifest) + } + + /// Whether the admission still holds the scope the session began under. + pub fn is_admitted(&self) -> bool { + self.held.guards(self.scope) + } +} diff --git a/crates/worldscript-secure-storage/src/lib.rs b/crates/worldscript-secure-storage/src/lib.rs index 88961ea43..806d4f741 100644 --- a/crates/worldscript-secure-storage/src/lib.rs +++ b/crates/worldscript-secure-storage/src/lib.rs @@ -25,6 +25,7 @@ pub mod anchor_codec; mod authority; pub mod catalog; pub mod commit; +pub mod conversion; pub mod disposition; pub mod durable; pub mod envelope; @@ -80,6 +81,7 @@ pub use commit::{ CommittedGeneration, Debris, DebrisKind, MarkerCommitted, Reconciled, RecordLocation, RecordStore, RecoveryReason, Resolution, WriteRequest, }; +pub use conversion::{begin_conversion, ConversionBegin, ConversionError, ConversionSession}; pub use disposition::{disposition, is_r15_record_class, Disposition}; pub use durable::{ generation_path, stage_and_promote, stage_and_promote_envelope, staging_path, diff --git a/crates/worldscript-secure-storage/tests/gate4d_conversion_entry_test.rs b/crates/worldscript-secure-storage/tests/gate4d_conversion_entry_test.rs new file mode 100644 index 000000000..100b65cd2 --- /dev/null +++ b/crates/worldscript-secure-storage/tests/gate4d_conversion_entry_test.rs @@ -0,0 +1,534 @@ +//! Gate 4D C2a: the entry gate of the exclusive conversion driver (§10.3 `ADMIT`, `CONVERT`). The gate +//! needs the installation's exclusive admission, re-reads the journal the committed root vouches for +//! and refuses anything but the owner's conversion; it writes nothing. Every scenario therefore +//! commits a real root that binds a real journal. + +use std::collections::BTreeMap; +use std::ffi::OsString; +use std::fs; +use std::fs::File; +use std::io; +use std::path::{Path, PathBuf}; +use std::sync::atomic::{AtomicU32, Ordering}; + +use worldscript_secure_storage::memory_provider::MemoryKeyProvider; +use worldscript_secure_storage::{ + begin_conversion, commit_catalog_change, commit_root, content_digest, empty_inventory_digest, + empty_journal_page_set_digest, generation_path, load_catalog, operation_type, phase_code, + write_key_epoch, AdmissionScope, AuthorityError, CatalogChange, CatalogCommit, ConversionBegin, + ConversionError, DirectoryDurability, DurableFs, ExclusiveAdmissionGuard, InstallationScopeId, + JournalDurableError, JournalManifest, JournalRouteError, JournalSource, Key, KeyEpochCommit, + KeyEpochRecord, KeyEpochStatus, KeyProvider, LiveMigration, MigrationExecutionError, + MigrationFence, RecordClass, RecordIdentity, RecordMeta, RootBody, RootCommitEvidence, + RootCommitGuard, RootCommitRequest, RootCommitState, RootKeyRefV1, RootLayout, StdFs, + WriteOperationId, JOURNAL_MANIFEST_RECORD_SCHEMA, OPERATION_ADMISSION_LOCK_FILE, +}; + +const OPERATION: &str = "conversion-op"; +const OWNER: &str = "owner-a"; +const REVISION: u64 = 3; +const FENCE: u64 = 4; +/// A lease long past its expiry: the gate reads no clock, so it is still the owner's. +const LEASE_EXPIRY: u64 = 1_000; +const MATERIAL: [u8; 32] = [9; 32]; +const EPOCH: u64 = 1; + +/// An `ENABLE` manifest at the bound revision, owned by [`OWNER`], in `phase`. The flag follows the +/// phase: forbidden before `ADMIT`, required from `CONVERT` to `DONE`, free for `ADMIT` and +/// `RECOVERY_REQUIRED`, where it is set here. +fn manifest(phase: u32) -> JournalManifest { + let before_admit = phase < phase_code::ADMIT; + JournalManifest { + operation_id: OPERATION.into(), + journal_revision: REVISION, + operation_type: operation_type::ENABLE, + phase, + source_epoch: 0, + target_epoch: EPOCH, + has_target_root_key_ref: true, + target_root_key_ref_digest: Some([0x42; 32]), + fencing_generation: FENCE, + inventory_version: 1, + inventory_digest: empty_inventory_digest(1), + page_count: 0, + entry_count: 0, + journal_page_set_digest: empty_journal_page_set_digest(), + final_inventory_captured: !before_admit, + cursor_page_index: 0, + cursor_entry_index: 0, + has_lease_owner: true, + lease_owner_id: Some(OWNER.into()), + lease_expires_unix_ms: Some(LEASE_EXPIRY), + recovery_reason_code: u32::from(phase == phase_code::RECOVERY_REQUIRED), + } +} + +fn journal_key() -> Key { + Key::from_bytes(&mut MATERIAL.clone()) +} + +/// A directory this test created; removed when dropped (`create_dir` fails on an existing path, so a +/// leftover of an earlier process is skipped, never reused or removed). +struct Dir(PathBuf); + +impl Dir { + fn new() -> Self { + static NEXT: AtomicU32 = AtomicU32::new(0); + loop { + let path = std::env::temp_dir().join(format!( + "wss-gate4d-conversion-{}-{}", + std::process::id(), + NEXT.fetch_add(1, Ordering::Relaxed) + )); + match fs::create_dir(&path) { + Ok(()) => return Dir(path), + Err(error) if error.kind() == io::ErrorKind::AlreadyExists => {} + Err(error) => panic!("cannot create {}: {error}", path.display()), + } + } + } +} + +impl Drop for Dir { + fn drop(&mut self) { + let _ = fs::remove_dir_all(&self.0); + } +} + +/// An installation directory holding the authority root and a journal, and a provider with the +/// epoch-1 key: the root's registry, its first root and (once bound) a root binding the journal. +struct Fixture { + base: Dir, + provider: MemoryKeyProvider, + scope: InstallationScopeId, + key_ref: RootKeyRefV1, +} + +impl Fixture { + /// The journal stored as the root-named generation of `manifest` and a root that does not yet + /// bind it; returns the binding that names it. + fn unbound(manifest: &JournalManifest) -> (Self, LiveMigration) { + let base = Dir::new(); + for slot in ["authority/slot-a", "authority/slot-b", "journal"] { + fs::create_dir_all(base.0.join(slot)).unwrap(); + } + let mut provider = MemoryKeyProvider::new(); + let scope = provider.read_or_provision_installation_scope().unwrap(); + let key_ref = provider.import_epoch_key(EPOCH, MATERIAL).unwrap(); + provider.unlock().unwrap(); + let mut fixture = Fixture { + base, + provider, + scope, + key_ref, + }; + fixture.register_epoch(); + let live = fixture.store_journal(manifest); + fixture.commit_first_root(); + (fixture, live) + } + + /// As [`Fixture::unbound`], with the root binding the journal. + fn bound(manifest: &JournalManifest) -> Self { + let (mut fixture, live) = Fixture::unbound(manifest); + fixture.bind(&live); + fixture + } + + fn root_dir(&self) -> PathBuf { + self.base.0.join("authority") + } + + fn journal_dir(&self) -> PathBuf { + self.base.0.join("journal") + } + + fn register_epoch(&self) { + let record = KeyEpochRecord { + epoch: EPOCH, + status: KeyEpochStatus::Active, + root_key_ref: self.key_ref.clone(), + }; + let root_dir = self.root_dir(); + let guard = RootCommitGuard::acquire(&root_dir).unwrap(); + let commit = KeyEpochCommit { + scope: &self.scope, + record: &record, + registry_generation: 1, + root_key_ref: &self.key_ref, + key_epoch: EPOCH, + held: &guard, + }; + let layout = RootLayout { + root_dir: &root_dir, + }; + write_key_epoch(&mut StdFs, &self.provider, layout, commit).unwrap(); + } + + fn store_journal(&self, manifest: &JournalManifest) -> LiveMigration { + let identity = RecordIdentity::new(RecordClass::Migration, &[OPERATION]).unwrap(); + let meta = RecordMeta { + key_epoch: EPOCH, + record_generation: REVISION, + record_schema: JOURNAL_MANIFEST_RECORD_SCHEMA, + }; + let sealed = manifest.seal(&journal_key(), &identity, meta).unwrap(); + fs::write(generation_path(&self.journal_dir(), REVISION), &sealed).unwrap(); + LiveMigration { + operation_id: OPERATION.into(), + fencing_generation: FENCE, + journal_revision: REVISION, + manifest_digest: content_digest(&sealed), + } + } + + fn commit_first_root(&mut self) { + let root_dir = self.root_dir(); + let commit = CatalogCommit { + change: CatalogChange { + upsert: &[], + remove: &[], + }, + root_key_ref: &self.key_ref, + active_key_epoch: EPOCH, + operation_id: "first-root", + }; + let layout = RootLayout { + root_dir: &root_dir, + }; + commit_catalog_change(&mut StdFs, &mut self.provider, layout, commit).unwrap(); + } + + /// The next root, binding `live`. No producer of a bind exists in the crate yet, so the body is + /// committed directly. + fn bind(&mut self, live: &LiveMigration) { + let root_dir = self.root_dir(); + let layout = RootLayout { + root_dir: &root_dir, + }; + let catalog = load_catalog(&mut StdFs, &self.provider, layout) + .unwrap() + .unwrap(); + let root = RootBody { + root_generation: catalog.root.root_generation + 1, + commit_evidence: RootCommitEvidence { + operation_id: live.operation_id.clone(), + fencing_generation: live.fencing_generation, + state: RootCommitState::Committed, + }, + live_migration: Some(live.clone()), + ..catalog.root + }; + let guard = RootCommitGuard::acquire(&root_dir).unwrap(); + let request = RootCommitRequest { + scope: &self.scope, + root: &root, + root_key_ref: &self.key_ref, + held: &guard, + }; + commit_root(&mut StdFs, &mut self.provider, layout, request).unwrap(); + } + + /// The locations a gate request names, with a write operation of its own. + fn paths(&self) -> Paths { + Paths { + installation: self.base.0.clone(), + root: self.root_dir(), + journal: self.journal_dir(), + operation: WriteOperationId::generate().unwrap(), + } + } + + /// The gate as `owner` under this installation's own admission, over the real file system. + fn begin(&self, owner: &str) -> Result<(), ConversionError> { + let paths = self.paths(); + let mut held = paths.admission(); + begin_conversion(&mut held, &mut StdFs, &self.provider, paths.begin(owner)).map(|_| ()) + } + + /// Every file under the fixture with a digest of its content. + fn snapshot(&self) -> BTreeMap { + let mut files = BTreeMap::new(); + collect(&self.base.0, &self.base.0, &mut files); + files + } +} + +/// The directories and the write operation a gate request refers to. +struct Paths { + installation: PathBuf, + root: PathBuf, + journal: PathBuf, + operation: WriteOperationId, +} + +impl Paths { + fn scope(&self) -> AdmissionScope<'_> { + AdmissionScope { + installation_dir: &self.installation, + root_dir: &self.root, + } + } + + /// The exclusive admission of the installation these paths name. + fn admission(&self) -> ExclusiveAdmissionGuard { + ExclusiveAdmissionGuard::try_acquire(self.scope()) + .unwrap() + .expect("nothing else holds this installation") + } + + fn begin<'a>(&'a self, owner: &'a str) -> ConversionBegin<'a> { + ConversionBegin { + scope: self.scope(), + journal: JournalSource { + dir: &self.journal, + operation: &self.operation, + }, + owner_id: owner, + } + } +} + +/// The real file system, reporting to `on_read` after every read has happened. +struct WatchedFs { + on_read: H, +} + +impl DurableFs for WatchedFs { + type File = File; + + fn create_new(&mut self, path: &Path) -> io::Result { + StdFs.create_new(path) + } + + fn sync_file(&mut self, file: &mut File) -> io::Result<()> { + StdFs.sync_file(file) + } + + fn read(&mut self, path: &Path) -> io::Result> { + let bytes = StdFs.read(path); + (self.on_read)(path); + bytes + } + + fn read_at_most(&mut self, path: &Path, limit: usize) -> io::Result>> { + let bytes = StdFs.read_at_most(path, limit); + (self.on_read)(path); + bytes + } + + fn link_no_replace(&mut self, from: &Path, to: &Path) -> io::Result<()> { + StdFs.link_no_replace(from, to) + } + + fn remove_file(&mut self, path: &Path) -> io::Result<()> { + StdFs.remove_file(path) + } + + fn sync_dir(&mut self, dir: &Path) -> io::Result { + StdFs.sync_dir(dir) + } + + fn list_dir(&mut self, dir: &Path) -> io::Result> { + StdFs.list_dir(dir) + } + + fn rename_replace(&mut self, from: &Path, to: &Path) -> io::Result<()> { + StdFs.rename_replace(from, to) + } + + fn create_dir_all(&mut self, dir: &Path) -> io::Result<()> { + StdFs.create_dir_all(dir) + } +} + +/// Collects every file below `dir` except the admission coordination file, which Windows creates when +/// an admission is acquired and whose bytes have no authority: the gate must not change anything else. +fn collect(base: &Path, dir: &Path, files: &mut BTreeMap) { + for entry in fs::read_dir(dir).unwrap() { + let path = entry.unwrap().path(); + if path.is_dir() { + collect(base, &path, files); + } else if path + .file_name() + .is_some_and(|name| name != OPERATION_ADMISSION_LOCK_FILE) + { + let relative = path.strip_prefix(base).unwrap().to_path_buf(); + files.insert(relative, content_digest(&fs::read(&path).unwrap())); + } + } +} + +#[test] +fn the_owner_in_admit_or_convert_gets_the_committed_state_and_nothing_is_written() { + let mut seen = Vec::new(); + for phase in [phase_code::ADMIT, phase_code::CONVERT] { + let committed = manifest(phase); + let fixture = Fixture::bound(&committed); + let before = fixture.snapshot(); + let paths = fixture.paths(); + let mut held = paths.admission(); + let session = + begin_conversion(&mut held, &mut StdFs, &fixture.provider, paths.begin(OWNER)).unwrap(); + seen.push(( + session.manifest() == &committed, + session.fence() == MigrationFence::from_manifest(&committed), + session.live().journal_revision == REVISION && session.is_admitted(), + fixture.snapshot() == before, + )); + } + assert_eq!(seen, vec![(true, true, true, true); 2]); +} + +#[test] +fn every_other_phase_is_refused_with_its_own_reason() { + let phases = [ + phase_code::BOOTSTRAP_TARGET, + phase_code::DISCOVER, + phase_code::PREPARE, + phase_code::VERIFY, + phase_code::COMMIT, + phase_code::RETIRE_OLD_AUTHORITY, + phase_code::FINALIZE, + phase_code::DONE, + phase_code::RECOVERY_REQUIRED, + ]; + let outcomes: Vec<_> = phases + .iter() + .map(|&phase| Fixture::bound(&manifest(phase)).begin(OWNER).unwrap_err()) + .collect(); + let mut expected = vec![ConversionError::PhaseNotConvertible; 7]; + expected.extend([ + ConversionError::TerminalPhase, + ConversionError::RecoveryRequired, + ]); + assert_eq!(outcomes, expected); +} + +#[test] +fn an_admit_manifest_without_the_final_inventory_is_refused() { + let mut open = manifest(phase_code::ADMIT); + open.final_inventory_captured = false; + let fixture = Fixture::bound(&open); + let before = fixture.snapshot(); + assert_eq!( + fixture.begin(OWNER), + Err(ConversionError::FinalInventoryNotCaptured) + ); + assert_eq!(fixture.snapshot(), before); +} + +#[test] +fn a_lease_of_another_owner_or_of_none_is_refused() { + let mut unowned = manifest(phase_code::ADMIT); + unowned.has_lease_owner = false; + unowned.lease_owner_id = None; + unowned.lease_expires_unix_ms = None; + let outcomes = vec![ + Fixture::bound(&manifest(phase_code::ADMIT)).begin("owner-b"), + Fixture::bound(&manifest(phase_code::CONVERT)).begin(""), + Fixture::bound(&unowned).begin(OWNER), + ]; + assert_eq!(outcomes, vec![Err(ConversionError::ForeignLeaseOwner); 3]); +} + +/// The gate over `fixture` as its owner under `held`, reporting every read to `on_read`. +fn begin_watched( + fixture: &Fixture, + held: &mut ExclusiveAdmissionGuard, + on_read: H, +) -> Result<(), ConversionError> { + let paths = fixture.paths(); + let mut watched = WatchedFs { on_read }; + begin_conversion(held, &mut watched, &fixture.provider, paths.begin(OWNER)).map(|_| ()) +} + +#[test] +fn a_guard_admitted_for_another_installation_is_refused_before_anything_is_read() { + let committed = manifest(phase_code::ADMIT); + let (fixture, other) = (Fixture::bound(&committed), Fixture::bound(&committed)); + let before = fixture.snapshot(); + let mut reads = 0; + let outcome = begin_watched(&fixture, &mut other.paths().admission(), |_| reads += 1); + let unchanged = fixture.snapshot() == before; + assert_eq!( + (outcome, reads, unchanged), + (Err(ConversionError::NotAdmitted), 0, true) + ); +} + +#[cfg(unix)] +#[test] +fn an_installation_replaced_while_the_journal_is_read_is_refused() { + // The gate is deterministic, so a first run counts its reads and a second run moves the + // installation directory away right after the last one: only the check after the reads sees it. + let committed = manifest(phase_code::ADMIT); + let counted = Fixture::bound(&committed); + let mut total = 0; + begin_watched(&counted, &mut counted.paths().admission(), |_| total += 1).unwrap(); + let fixture = Fixture::bound(&committed); + let installation = fixture.base.0.clone(); + let moved = installation.with_extension("moved"); + let mut seen = 0; + let outcome = begin_watched(&fixture, &mut fixture.paths().admission(), |_| { + seen += 1; + if seen == total { + fs::rename(&installation, &moved).unwrap(); + } + }); + fs::rename(&moved, &installation).unwrap(); + assert_eq!(outcome, Err(ConversionError::NotAdmitted)); +} + +#[test] +fn a_root_that_binds_no_migration_is_refused() { + let (fixture, _live) = Fixture::unbound(&manifest(phase_code::ADMIT)); + assert_eq!( + fixture.begin(OWNER), + Err(ConversionError::Authority(AuthorityError::NoLiveMigration)) + ); +} + +#[test] +fn a_journal_generation_the_root_does_not_name_is_refused() { + let fixture = Fixture::bound(&manifest(phase_code::ADMIT)); + let mut tampered = manifest(phase_code::ADMIT); + tampered.lease_owner_id = Some("someone-else".into()); + fixture.store_journal(&tampered); + assert_eq!( + fixture.begin("someone-else"), + Err(ConversionError::Authority(AuthorityError::JournalRoute( + JournalRouteError::Journal(JournalDurableError::Authority( + MigrationExecutionError::LiveBindingMismatch + )) + ))) + ); +} + +#[test] +fn a_sibling_generation_is_never_read_instead_of_the_root_named_one() { + // A newer generation in the journal directory is not authority: the gate reads the one the root + // names, so an unrelated later file neither replaces it nor makes the gate refuse. + let committed = manifest(phase_code::ADMIT); + let fixture = Fixture::bound(&committed); + let mut later = manifest(phase_code::CONVERT); + later.journal_revision = REVISION + 1; + later.lease_owner_id = Some("someone-else".into()); + let identity = RecordIdentity::new(RecordClass::Migration, &[OPERATION]).unwrap(); + let meta = RecordMeta { + key_epoch: EPOCH, + record_generation: later.journal_revision, + record_schema: JOURNAL_MANIFEST_RECORD_SCHEMA, + }; + let sealed = later.seal(&journal_key(), &identity, meta).unwrap(); + fs::write( + generation_path(&fixture.journal_dir(), REVISION + 1), + sealed, + ) + .unwrap(); + assert_eq!(fixture.begin(OWNER), Ok(())); + assert_eq!( + fixture.begin("someone-else"), + Err(ConversionError::ForeignLeaseOwner) + ); +} diff --git a/crates/worldscript-secure-storage/tests/gate4d_journal_route_test.rs b/crates/worldscript-secure-storage/tests/gate4d_journal_route_test.rs index 267baaae9..e3e149d83 100644 --- a/crates/worldscript-secure-storage/tests/gate4d_journal_route_test.rs +++ b/crates/worldscript-secure-storage/tests/gate4d_journal_route_test.rs @@ -4,7 +4,7 @@ //! root's active epoch and never through anything the caller supplies. Every scenario therefore //! commits a real root that binds the journal. Slice D2b-3b: the four composed journal operations //! use that route, so an unroutable epoch refuses each of them before a journal write. Slice D3b: so -//! does the renewal of the owner's lease. +//! does the renewal of the owner's lease. Slice C2a: so does the entry gate of the conversion driver. use std::collections::BTreeMap; use std::fs; @@ -14,20 +14,21 @@ use std::sync::atomic::{AtomicU32, Ordering}; use worldscript_secure_storage::memory_provider::MemoryKeyProvider; use worldscript_secure_storage::{ - advance_live_migration, begin_streamed_capture, commit_catalog_change, + advance_live_migration, begin_conversion, begin_streamed_capture, commit_catalog_change, commit_inventory_capture, commit_journal_checkpoint, commit_journal_takeover, commit_lease_renewal, commit_root, commit_streamed_inventory_capture, content_digest, empty_inventory_digest, empty_journal_page_set_digest, generation_path, load_catalog, - operation_type, phase_code, resolve_journal_key, write_key_epoch, AuthorityError, - BindingAdvance, CandidateConflict, CaptureStart, CatalogChange, CatalogCommit, - InstallationScopeId, InventoryCapture, JournalCheckpoint, JournalDurableContext, - JournalDurableError, JournalError, JournalManifest, JournalRoute, JournalRouteError, - JournalSource, JournalTakeoverCommit, Key, KeyEpochCommit, KeyEpochRecord, KeyEpochStatus, - KeyProvider, LiveMigration, ManifestRead, MigrationExecutionError, MigrationFence, OpenError, - RecordClass, RecordIdentity, RecordMeta, RootBody, RootCommitEvidence, RootCommitGuard, - RootCommitRequest, RootCommitState, RootKeyRefV1, RootLayout, SessionBegin, StagedCapture, - StdFs, StreamedCapture, StreamedInventoryCapture, WriteOperationId, - JOURNAL_MANIFEST_RECORD_SCHEMA, + operation_type, phase_code, resolve_journal_key, write_key_epoch, AdmissionScope, + AuthorityError, BindingAdvance, CandidateConflict, CaptureStart, CatalogChange, CatalogCommit, + ConversionBegin, ConversionError, ExclusiveAdmissionGuard, InstallationScopeId, + InventoryCapture, JournalCheckpoint, JournalDurableContext, JournalDurableError, JournalError, + JournalManifest, JournalRoute, JournalRouteError, JournalSource, JournalTakeoverCommit, Key, + KeyEpochCommit, KeyEpochRecord, KeyEpochStatus, KeyProvider, LiveMigration, ManifestRead, + MigrationExecutionError, MigrationFence, OpenError, RecordClass, RecordIdentity, RecordMeta, + RootBody, RootCommitEvidence, RootCommitGuard, RootCommitRequest, RootCommitState, + RootKeyRefV1, RootLayout, SessionBegin, StagedCapture, StdFs, StreamedCapture, + StreamedInventoryCapture, WriteOperationId, JOURNAL_MANIFEST_RECORD_SCHEMA, + OPERATION_ADMISSION_LOCK_FILE, }; const OPERATION: &str = "route-op"; @@ -251,12 +252,17 @@ impl Fixture { } } +/// Collects every file below `dir` except the admission coordination file, which Windows creates when +/// an admission is acquired and whose bytes have no authority. fn collect(base: &Path, dir: &Path, files: &mut BTreeMap) { for entry in fs::read_dir(dir).unwrap() { let path = entry.unwrap().path(); if path.is_dir() { collect(base, &path, files); - } else { + } else if path + .file_name() + .is_some_and(|name| name != OPERATION_ADMISSION_LOCK_FILE) + { let relative = path.strip_prefix(base).unwrap().to_path_buf(); files.insert(relative, content_digest(&fs::read(&path).unwrap())); } @@ -599,6 +605,32 @@ impl Fixture { begin_streamed_capture(&mut StdFs, &self.provider, layout, begin).unwrap_err() } + /// The entry gate of the conversion driver over a valid scope and owner, which is expected to be + /// refused: the key is routed from the root before the manifest is loaded. + fn conversion_is_refused(&self) -> AuthorityError { + let (root_dir, journal_dir) = (self.root_dir(), self.journal_dir()); + let scope = AdmissionScope { + installation_dir: &self.base.0, + root_dir: &root_dir, + }; + let mut held = ExclusiveAdmissionGuard::try_acquire(scope) + .unwrap() + .expect("nothing else holds this installation"); + let operation = WriteOperationId::generate().unwrap(); + let begin = ConversionBegin { + scope, + journal: JournalSource { + dir: &journal_dir, + operation: &operation, + }, + owner_id: "owner-a", + }; + match begin_conversion(&mut held, &mut StdFs, &self.provider, begin).unwrap_err() { + ConversionError::Authority(error) => error, + other => panic!("the route must refuse first, not {other:?}"), + } + } + /// The five composed journal operations over valid inputs for each (a checkpoint and a capture of /// the next revision, a takeover claim by a new owner at fence plus one, an advance to the stored /// successor, a renewal of the owner's lease): each is expected to be refused, and the error of @@ -693,6 +725,7 @@ fn a_revoked_journal_epoch_refuses_every_journal_operation_before_a_write() { let mut errors = fixture.every_operation_is_refused(&stored); errors.push(fixture.streamed_capture_is_refused(&staged)); errors.push(fixture.session_is_refused()); + errors.push(fixture.conversion_is_refused()); for error in errors { assert_eq!( error, @@ -713,6 +746,7 @@ fn an_unregistered_journal_epoch_refuses_every_journal_operation_before_a_write( let mut errors = fixture.every_operation_is_refused(&stored); errors.push(fixture.streamed_capture_is_refused(&staged)); errors.push(fixture.session_is_refused()); + errors.push(fixture.conversion_is_refused()); for error in errors { assert_eq!( error, @@ -737,6 +771,7 @@ fn a_route_to_another_key_is_refused_by_the_authenticated_load_before_a_write() let mut errors = fixture.every_operation_is_refused(&stored); errors.push(fixture.streamed_capture_is_refused(&staged)); errors.push(fixture.session_is_refused()); + errors.push(fixture.conversion_is_refused()); for error in errors { assert_eq!( error, diff --git a/docs/native/R15-SECURE-STORAGE-CONTRACT.md b/docs/native/R15-SECURE-STORAGE-CONTRACT.md index aa8388070..cc0874f27 100644 --- a/docs/native/R15-SECURE-STORAGE-CONTRACT.md +++ b/docs/native/R15-SECURE-STORAGE-CONTRACT.md @@ -2377,6 +2377,17 @@ nor the plain binding advance can carry a lease change, so this is the only path extended. A renewal and a takeover that race for the same revision are settled by the root lock: whichever commits first makes the other stale, because a takeover advances the fence and a renewal advances the expiry the takeover required to have lapsed. + +**The conversion entry gate.** Conversion runs under the installation's exclusive operation admission +(§11), taken by mutable borrow so the admission outlives the conversion session by construction, and over +the root of the scope that admission was acquired for: the root layout is derived from that scope, never a +second input. The gate re-reads what the committed root vouches for instead of trusting a caller's copy (the +binding from the root, the journal key from the key-epoch registry, the manifest by the exact root-named +path authenticated against the binding's digest) and refuses unless the operation is in `ADMIT` (conversion +can be entered) or `CONVERT` (a resumed conversion), the final inventory was captured, and the committed +lease belongs to the caller. A lease owned by another owner, or by none, is refused: the gate takes nothing +over, and a restart is the takeover followed by a fresh begin. The gate reads no clock, writes nothing and +holds no key, and the identity of the admitted directories is checked before the reads and again after them. A token check performed as a separate preflight is insufficient. Every mutation-capable adapter operation therefore exposes the semantic equivalent of `with_fence(fencing_generation, mutation_and_durability)`: it acquires the cross-process migration diff --git a/docs/native/r15/GATE4D-JOURNAL-DURABLE-EVIDENCE.md b/docs/native/r15/GATE4D-JOURNAL-DURABLE-EVIDENCE.md index e0fe37dd5..2bd45a118 100644 --- a/docs/native/r15/GATE4D-JOURNAL-DURABLE-EVIDENCE.md +++ b/docs/native/r15/GATE4D-JOURNAL-DURABLE-EVIDENCE.md @@ -665,6 +665,40 @@ after authentication (D2b-2), and resolving the journal key through the authenti readability proof for a root already at the target epoch (D2b-3a), used by every journal-owner operation (D2b-3b). Each is described in its own section below. +## Slice C2a — the entry gate of the exclusive conversion driver + +The journal-owner operations take no admission guard, and nothing read the committed journal state without a +key in the caller's hands. `begin_conversion` is the first part of the conversion driver (maintainer decision +D): it makes exclusive admission structural and re-reads the authenticated root-bound manifest, and it +writes nothing. + +| Step | Rule | +|---|---| +| input | `&mut ExclusiveAdmissionGuard` (a shared guard does not type-check: compile_fail doctest) and `ConversionBegin{scope, journal, owner_id}`; the root layout is derived from `scope`, so the root cannot differ from the one admitted | +| admission | `guards(scope)` is checked before anything is read and again after the reads; a guard for another installation is `NotAdmitted` with no read made, and an installation moved away during the reads is `NotAdmitted` | +| read | `read_committed_journal`: the binding from the committed root alone (`NoLiveMigration` if none), the key through the registry (`JournalRoute` errors), the manifest by the exact root-named path authenticated against the binding digest; no sibling generation is read | +| gate | phase `ADMIT` or `CONVERT` (`TerminalPhase` for `DONE`, `RecoveryRequired` for `RECOVERY_REQUIRED`, `PhaseNotConvertible` otherwise), `final_inventory_captured` (`FinalInventoryNotCaptured`), and the committed lease owner equal to the caller (`ForeignLeaseOwner`; an absent lease is foreign) | +| result | a `ConversionSession` that keeps the `&mut` borrow of the admission (one session per admission) and exposes the manifest, the binding and the fence it read; it holds no key | + +Decisions, disclosed: (a) three slices for the driver, this one mutating nothing so that the exclusivity proof +lands before any conversion write; (b) the layout is derived from the admitted scope instead of being a second +parameter; (c) a foreign lease owner is refused rather than taken over, and whether exclusive admission may +replace lease expiry as the proof that a former owner is gone is a normative question left open (the +cross-process lease CAS stays residual); (d) no clock is read, so a lease past its expiry is still the +owner's while nobody has taken over; (e) the journal source's operation is a write-operation identifier for +staging names, not the migration operation, so an operation mismatch is not a separate refusal here: the +manifest is opened under the binding's operation identity and the binding vouches for it. + +Proof (`gate4d_conversion_entry_test`): the owner in `ADMIT` and in `CONVERT` gets exactly the committed +manifest, fence and binding with every file of the installation unchanged; every other phase is refused with its +own reason; an `ADMIT` manifest without the final inventory, a lease of another owner, of the empty owner and +of none, a root that binds no migration and a root-named generation that is not the bound bytes are refused; +a guard for another installation is refused before any read; an installation moved away during the reads is +refused; a newer sibling generation is never read. The key route refuses the gate with a revoked or unregistered +epoch and with a route to another key, beside the other journal-owner operations +(`gate4d_journal_route_test`). Mutation-checked: each of the two admission checks, each phase arm, the final +inventory requirement and the owner check, removed one at a time, fails the test that owns it. + ## Streaming capture, stage two (b) — the staging session routes the key After stage two (a) the stage-one builder still took the journal key, the committed manifest and the binding @@ -987,7 +1021,7 @@ this API's reach; the journal key route of D2b-3 fails closed on a `Revoked` or reads still use the whole-file `DurableFs::read`. Applying the same size limits to them is a separate slice, recorded as an acceptance criterion on #359. - A caller of the lease renewal: `commit_lease_renewal` exists (D3b) but no orchestrator renews a lease yet. -- Conversion (C2+) over the verified page set: it must require exclusive admission by construction and re-read the root-bound manifest with `final_inventory_captured = 1` (maintainer decision D). +- Conversion over the verified page set: the entry gate exists (C2a: exclusive admission by construction, the root-bound manifest re-read, `final_inventory_captured = 1`; maintainer decision D), but nothing yet moves the journal under it: entering `CONVERT`, advancing the cursor and renewing the lease through the gate (C2b), the cursor-driven iteration with a per-entry step and the crash/resume evidence (C2c). - Write barrier of the final capture: `commit_inventory_capture` takes no admission guard; the barrier is the durable `ADMIT` phase the orchestrator establishes by draining writers, and the write path must refuse ordinary mutating writes by that phase (`ordinary_mutating_writes_admitted`) before the final capture has a caller (Gate 4E/5; acceptance criterion on #359). - Inheriting unchanged pages: the C1b-2 reader now returns the authenticated page references, so the store may accept a page that keeps an earlier generation if those references name exactly its bytes (acceptance criterion on #359, a follow-up slice). Until then every page of a capture is rewritten at the new revision. - Reclaiming abandoned pending directories: an attempt that was killed, or a finished capture dropped without `discard`, leaves inert files under `inventory/pending-*`, and every attempt leaves its empty page directories; none is authority or ever read. Reclaiming them needs a directory-removal primitive and a sweep that knows no live attempt owns them, as for the orphaned digest directories of a discarded capture (acceptance criterion on #359). @@ -1009,6 +1043,8 @@ Integration: `crates/worldscript-secure-storage/tests/gate4d_journal_durable_tes Integration: `crates/worldscript-secure-storage/tests/gate4d_root_binding_test.rs` (9 B2b-1 binding-advance cases, 6 B2b-2 checkpoint cases, 5 B2b-3 candidate-retry cases, 4 B2b-4 successor cases and 6 B2c takeover cases and 7 B2d discard cases). Unit: `crates/worldscript-secure-storage/tests/gate4d_migration_state_test.rs` (8 B2b-4 successor-relation cases and 5 B2c takeover cases beside the earlier state-machine cases). +Integration: `crates/worldscript-secure-storage/tests/gate4d_conversion_entry_test.rs` (the conversion entry gate), and the gate as a further operation in `gate4d_journal_route_test.rs`. + Unit (mutex): `journal_durable_mutex_blocks_try_lock_while_guard_held` and `with_fence_holds_mutex_during_closure` in `journal::durable::mutex_proof`. diff --git a/docs/native/r15/GATE4D-SLICE-B-GAP-MATRIX.md b/docs/native/r15/GATE4D-SLICE-B-GAP-MATRIX.md index 9083de979..cc1cc26c4 100644 --- a/docs/native/r15/GATE4D-SLICE-B-GAP-MATRIX.md +++ b/docs/native/r15/GATE4D-SLICE-B-GAP-MATRIX.md @@ -92,6 +92,7 @@ load of the newer generation yet). - Streaming capture, stage one: `StreamedCapture` builds and stages a captured inventory one page at a time (the total announced up front, because the inventory digest commits to it first), each page sealed once and its envelope staged in a private `inventory/pending-*` directory that is never authority; the successor is built by the same function as `capture_inventory`; `load_staged_page` confirms a staged page against its reference and treats a missing one as a broken attempt, not a recovery state. A capture is bound to its journal directory and key, and a handled failure removes the pages staged so far. Promotion and the composed commit are stage two (a), the key-routing session stage two (b). - Streaming capture, stage two (a): `promote_staged_inventory_fenced` stores the staged pages under the digest directory of the successor one page at a time (owner, root-named predecessor and capture successor refused before any write; each page digest-checked, absorbed into the inventory digest, stored with adoption of an identical page; the inventory digest confirmed) and `commit_streamed_inventory_capture` publishes the successor and advances the binding in `commit_inventory_capture`'s order through a shared core, removing the staged files only after the root has committed so a retry still has them. The key-routing staging session is stage two (b). - Streaming capture, stage two (b): `begin_streamed_capture` returns a `StagingSession` that routes the journal key from the root, reads the binding and the committed manifest from it and holds the journal directory once, so staging handles no key; `commit_streamed_inventory_capture` loses its committed-manifest and journal-directory inputs (it promotes into the staged directory and loads the manifest from the root under the lock). +- Conversion driver, slice C2a: `begin_conversion` takes the installation's exclusive admission by mutable borrow, derives the root layout from the admitted scope, re-reads the committed binding, the registry-routed key and the root-named manifest, and returns a `ConversionSession` only for the owner of the lease in `ADMIT` or `CONVERT` with the final inventory captured; it writes nothing. The steps that move the journal and the cursor-driven iteration are C2b and C2c. - Slice C: record conversion / mixed-key inventory execution as live truth requires - Gate 4E: first enable/disable closure - Root two-phase commit coupling with step F when journal + root must advance together (after B2)