status: preserve clean-status proofs across mixed Git writers - #21
Open
ttaylorr-oai wants to merge 129 commits into
Open
status: preserve clean-status proofs across mixed Git writers#21ttaylorr-oai wants to merge 129 commits into
ttaylorr-oai wants to merge 129 commits into
Conversation
GitHub exposes manual dispatch only for workflows present on the default branch. The controller itself must remain on the orphan meta branch so master stays identical to upstream. Add the fixed caller as a dedicated active topic. It delegates to the meta-pinned reusable workflow without putting controller code on codex.
The reusable controller reads its publication key from the codex-publish environment. The default-branch trampoline therefore needs only read access to Actions and repository contents; it no longer forwards repository secrets. Keep this exact workflow in the automation topic so the Actions page can dispatch the controller pinned to meta.
Codex rebuilds already call a trusted workflow on meta, but their dispatch-only trampoline cannot run pull request or merge-queue checks. Topics could therefore reach codex without verifying their review, branch ownership, or published base. Run the pinned admission workflow for pull requests targeting codex and for merge-group checks. Keep rebuild preparation limited to explicit workflow dispatch, and grant the admission job only read access.
The default-branch trampoline only listened for pull requests against codex and ran one admission job for every event. A preview pull request would therefore have no required check, and merge groups could not distinguish the production and preview lanes. Listen for both generated outputs, keep the existing production job and check context unchanged, and add a target-specific preview job that calls the trusted meta admission workflow with read-only permissions.
The release workflow cross-compiles Linux arm64 on an x64 runner and skips the smoke test for arm64 POSIX bundles. That prevents the workflow from executing the Linux artifact it just produced. Run Linux arm64 on GitHub's arm64 runner and install native development packages rather than configuring a foreign dpkg architecture. All matrix entries can then run the existing distribution smoke test.
Codex consumes Git release artifacts built with the Makefile's default -O2 flags. The release job compiles each artifact without link-time optimization. Add a release-only config.mak.openai and copy it into Git's ignored config.mak slot before building. Use thin LTO for Clang targets and automatic LTO for GCC targets, then check GIT-CFLAGS records the selected flag in every distribution job. Keeping the setting in config.mak.openai avoids carrying release-only policy in the upstream Makefile.
LTO can optimize across translation units, but the release job has no execution profile for the status, diff, clone, fetch, and repack paths Codex invokes frequently. Git's built-in profile target runs the 1,048-script test suite serially. That is too expensive for every release target and weights test-harness paths more heavily than the local workload. Extend config.mak.openai with GCC and LLVM profile modes. Gate GIT-CFLAGS on an instrumented build, run a short offline trainer, merge LLVM raw profiles when needed, and rebuild with profile-use flags. Each matrix entry runs on its target architecture, so it can execute the instrumented binary. Check that final GIT-CFLAGS includes a profile-use flag and increase the timeout for the second compilation pass. The focused trainer took about 30 seconds locally; the full macOS build/install validation completed with thin LTO and LLVM profile-use enabled.
Integrate the current tb/codex/automation topic into the internally distributed codex branch. Codex-Integration: tb/codex/automation@17738e2cc87ba67ed36cd1ffde983d43e01a5f41
Integrate the current tb/codex/geometric-maintenance-promisor topic into the internally distributed codex branch. Codex-Integration: tb/codex/geometric-maintenance-promisor@dc2fffc37cead551f8036c9ecab5e52a4cbee37b
Integrate the current tb/codex/release topic into the internally distributed codex branch. Codex-Integration: tb/codex/release@ba107e0ae8c7142238bb612e530d51d42f0280d3
Integrate the current dr/codex/dugite topic into the internally distributed codex branch. Codex-Integration: dr/codex/dugite@988cecced01f69765d599a2d6c023406af98fa1b
Integrate the current tb/codex/lto-pgo topic into the internally distributed codex branch. Codex-Integration: tb/codex/lto-pgo@88fcb4ac12c583bedf010e97ebf83cec240e3120
Clearing CE_FSMONITOR_VALID is not enough to make a provider event authoritative. With core.trustctime disabled, core.checkStat set to minimal, and a restored modification time, stat matching can still accept changed file contents. The same stale match can affect diff, apply, checkout, and unpack-trees. Mark a reported entry with the in-memory CE_CONTENT_CHECK_REQUIRED flag, clear CE_UPTODATE, and discard its cached stat data. Route diff, apply, checkout, and unpack-trees comparisons through ie_match_stat_with_content_check(), which calls ie_modified() only for marked non-gitlinks. Other direct ie_match_stat() callers retain their existing paths. Marking an entry up to date clears the transient flag. Ordinary entries, gitlinks, and unmarked zero-stat entries retain their existing stat behavior. Add hook regressions for restored timestamps, diff and status, indexed apply, checkout, case-insensitive unpacking, unchanged reset, and ordinary zero-stat behavior in t/t7519-status-fsmonitor.sh. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
A filesystem-monitor provider can know that its event history is incomplete without being able to identify every affected path. Treating such a response as an ordinary path leaves tracked entries, cached attributes, and untracked-cache state falsely valid. Reserve // as a provider-only global invalidation record. It cannot collide with a worktree-relative path. When the client receives it, discard cached attribute stacks and untracked-cache state, invalidate every tracked entry, and mark the fsmonitor extension changed. Recognize the existing trivial response only when a complete record consists of a single slash and NUL, newline, or carriage-return terminators. This prevents the new double-slash record from being discarded as a trivial response while preserving existing hook forms. Add a hook regression in t/t7519-status-fsmonitor.sh that changes a tracked file, restores its timestamp, emits the global marker, and requires status to report the change. Global invalidation intentionally scans the tracked index. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
An implicitly started fsmonitor daemon inherits its caller's repository environment and current directory. In a linked worktree, inherited Git directory, worktree, common-directory, prefix, and index settings can make the child discover a different repository than the worktree whose status requested the daemon. Resolve the requested worktree to its canonical path, start the child from that directory, and remove repository-addressing variables from its environment. Keep the existing daemon start command and return an error if the worktree cannot be resolved. Add a macOS regression that implicitly starts fsmonitor from a linked worktree and checks the daemon child's working directory in Trace2. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
A pathname monitor cannot establish that every name for a multiply-linked regular file lies inside its watch cone. Persisting CE_FSMONITOR_VALID after checking the tracked name can therefore hide a later write through an unmonitored hardlink. Use fsmonitor_stat_can_be_valid() to exclude regular files with more than one link from persistent fsmonitor validity when the platform reports real link counts. Apply that decision where index refresh, threaded preload, and diff-files first consume an actual stat. Preserve CE_UPTODATE for the current process and retain existing persistent validity for single-link and nonregular entries. Windows and Cygwin synthesize their link counts, so preserve their existing fsmonitor behavior without claiming the hardlink guarantee there. Add a hardlink regression in t/t7519-status-fsmonitor.sh on platforms with trustworthy stat metadata. It keeps a tracked hardlink outside the fsmonitor-valid bitmap and checks that a write through an alias outside the worktree appears in status. The deliberate cost is another stat in a subsequent process. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
Implicit fsmonitor startup resolves a Git command through the execution path and invokes its start subcommand. An overridden execution path can therefore select a different Git than the dispatcher that initiated the query, while adding another launcher between the client and daemon. Retain the absolute executable path during dispatcher initialization and expose it only for a real Git dispatcher. Start that executable directly with fsmonitor--daemon run --detach, then wait until its IPC socket is listening before accepting startup. Respect the configured startup timeout, defaulting to 60 seconds, and retain Git-command lookup when an authoritative dispatcher path is unavailable. The canonical worktree and sanitized environment established by S03/P01 remain in place. Update existing startup Trace2 checks for the direct invocation and add a macOS regression with a fake Git on the execution path to verify that the original executable is used. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
Darwin FSEvents identifies the pathname associated with a hardlink event, not every name referring to the same inode. Invalidating only that pathname can leave another tracked hardlink trusted after its contents change. Classify the event's absolute path before handling its hardlink flags. For worktree events, enqueue the provider-wide marker introduced by S04/P02 so clients content-check the tracked set. Leave gitdir events in the existing cookie and gitdir handling; otherwise reads of hardlinked object files could repeatedly trigger global invalidation. Add a MACOS,HARDLINKS daemon regression that rejects a marker for a gitdir hardlink, then verifies the marker and correct status for a changed worktree hardlink with its timestamp restored. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
The fsmonitor.startTimeout setting controls how long a client waits for daemon startup; the daemon's run subcommand does not consume it. Nevertheless, daemon configuration parsing validates that setting for every subcommand. A malformed value can consequently kill an implicitly started daemon before it opens its IPC socket. Pass a run-specific configuration flag into the callback and skip startup-timeout parsing only for run. Continue parsing other daemon settings normally, and preserve strict timeout validation for the explicit start subcommand. Add a macOS regression that verifies implicit status still starts the daemon with a malformed timeout while explicit daemon start rejects the same configuration. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
Changing a .gitattributes file can change how tracked content is converted without changing the tracked file's stat data. Invalidating the attribute-file path alone therefore leaves cached conversion state and affected fsmonitor-valid tracked entries falsely reusable. Recognize an exact .gitattributes basename in the refresh callback. Discard cached attribute stacks globally and strongly invalidate only tracked entries beneath that file's parent directory. A root attribute file invalidates all tracked entries; tracked entries in sibling directories remain valid after a nested attribute-file event. Mark the fsmonitor extension changed only when an entry is invalidated. Add Clar unit coverage for unrelated paths, nested-directory scope, root-directory scope, cleared validity, zeroed stat data, and the content-check marker. Register u-fsmonitor-attributes in both Makefile and t/meson.build so the suite is included in both build systems. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
An fsmonitor socket is selected through the Git directory, so separate worktree paths can reach the same daemon when they share that directory. A client in the second worktree can then consume change history from a daemon that watches the first, incorrectly treating changed files in its own worktree as clean. Hash the canonical worktree path together with its device and inode, plus birth time and generation on Apple platforms. Cache the resulting 64-character SHA-256 identity in the daemon and attach it to every client query. Check the identity before interpreting the requested token; reject missing or mismatched bindings with a cookie-synchronized trivial response that forces the ordinary refresh path. The protocol change must also tolerate a daemon left running by an older Git. Such a daemon treats a bound query as an opaque token and can return a plausible trivial response. After that exact response, query an unbound capability command. If the daemon does not advertise query-v1, serialize replacement through a per-socket restart lock, stop it, and start the invoking Git executable before retrying the bound query. Keep quit, flush, and capability control commands unbound. Bound daemon lifecycle retries, and fail the query instead of trusting history when the root cannot be identified or an incompatible daemon cannot be replaced. Regression tests cover shared-gitdir worktree aliases, replacement of a legacy daemon, and acceptance of a daemon that advertises a capability superset. The replacement test also verifies that the next status neither refreshes tracked entries nor starts another daemon. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
A provider can report a directory move or modification without naming a changed .gitattributes file beneath it. Existing directory handling invalidates tracked entries in the reported cone but can leave cached attribute stacks describing the old conversion rules. Discard cached attribute stacks only after directory handling matches at least one tracked index entry. Record semantic/attributes-cone with the number of matched entries. An unmatched directory keeps its existing case-correction and untracked-path fallback without speculatively flushing attribute state. Extend t/helper/test-read-cache.c to cache an old attribute, process a directory event, and require the new attribute value. Add hook regressions in t/t7519-status-fsmonitor.sh for both an indexed cone and an unmatched directory, including their distinct Trace2 behavior. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
load_index_extensions() enters its extension loop only when a complete eight-byte header fits before the trailing checksum. It nevertheless trusts the declared payload size. An oversized payload can send an extension parser beyond the mapped extension area, while an incomplete trailing header is silently ignored. Compute the checksum boundary once and require the initial offset, each complete header, and each declared payload to fit within it. Advance only by checked header and payload sizes, reject partial trailing headers, and report framing failures as index file corruption. Add a PERL_TEST_HELPERS regression test that overwrites an FSMN payload length with 0xffffffff and checks that porcelain-v2 status fails with the existing corruption diagnostic. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
read_one() allocates a subtree array even for leaf cache-tree nodes. It also inserts each serialized child through cache_tree_sub(), which searches children that the writer already emits in increasing order. Allocate a child array only for non-leaf nodes and append increasing child names directly. Retain subtree_nr + 2 pointer slots for each non-leaf, but allocate them without zeroing because only populated slots are inspected. Keep cache_tree_sub() as the compatibility fallback for older, unsorted input. Existing t/t0090-cache-tree.sh tests exercise ordinary cache-tree decoding. This change adds no dedicated unsorted-input regression or isolated benchmark. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
add_patterns() rejects pattern files larger than 100 MiB only after allocating and reading their complete contents. An oversized filesystem input can therefore exhaust the memory the limit is meant to protect, or terminate Git when GIT_ALLOC_LIMIT rejects the allocation. Check the size obtained from fstat() before allocating a filesystem pattern buffer. Preserve the existing warning, close the descriptor, and return the existing failure result. Keep the later size check for index-backed fallback data, whose size is unavailable before it is read. Strengthen the existing EXPENSIVE regression by reading its 101 MiB .gitignore under GIT_ALLOC_LIMIT=1m. The old ordering dies in xmallocz(); the early rejection preserves the expected warning without attempting the oversized allocation. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
The index extension worker decodes TREE and UNTR serially even though their parsers read the same immutable mapping and publish to different index_state fields. An unconditional additional worker would consume cache-entry workers and interfere with split-index assembly. Use the bounded framing from S02/P01 to select exactly one TREE and one UNTR extension. Require extension-offset metadata and at least four index workers; start an additional TREE worker only when both payloads reach 1 MiB. Leave at least two cache-entry workers available and join the TREE worker before unmapping the index. Keep LINK, duplicate or missing extensions, insufficient workers, small payloads, and auxiliary-worker creation failures on the existing serial path. Malformed framing still reports index file corruption. Allow GIT_TEST_PARALLEL_INDEX_EXTENSIONS to bypass only the payload threshold. Add a PTHREADS, UNTRACKED_CACHE, and SHA1 regression that compares parallel and serial status, cache-tree, and untracked-cache results and checks the extension/parallel/tree-untracked Trace2 marker. The regression unsets GIT_TEST_SPLIT_INDEX because split indexes intentionally remain on the serial path. The eligible path adds one auxiliary worker and its stack. The benchmark covers the complete series, not this patch in isolation. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
ttaylorr-oai
force-pushed
the
tb/codex/status-preview-unstable
branch
2 times, most recently
from
August 6, 2026 20:03
f26a55f to
c1113e5
Compare
A directory name observed during enumeration may resolve outside the original worktree after a rename, symlink replacement, magic-link traversal, or mount change. Path-based reopening would then inspect an unverified namespace. Introduce descriptor-relative Linux directory-open helpers. Prefer openat2() with beneath-root resolution and reject symlinks, magic links, and mount crossings when that syscall is available. Otherwise reject empty, absolute, dot-dot, and malformed paths. Open root-relative paths one component at a time and verify each mount. Keep direct child opens descriptor-relative; directory scanning verifies their mounts before enumeration. Register the opening module with Make, CMake, and Meson. The native Linux boundary build compiles it with DEVELOPER=1, but backend selection remains unchanged. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
A getdents64 record supplies a type hint, not proof that a path is a regular file or directory. Trusting that hint can hide a tracked replacement, misapply ignore rules, or report the wrong visible untracked shape. Parse record lengths and names within a bounded 1 MiB worker buffer. Require statx metadata and matching mount identity for tracked paths. When collecting untracked paths, obtain authoritative metadata before classifying a wholly untracked file or directory. Use directory hints only to schedule paths with tracked descendants. Preserve per-entry fallback for special and multiply linked tracked files. Stop an untracked subtree once its normal-status witness is visible, and invalidate uncertain untracked results. Register the module in all three Linux builds; the native DEVELOPER=1 boundary build compiles it without selecting the backend. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
Holding a directory descriptor establishes what workers read, but does not prove that the original directory stayed in the worktree. A child can also move under a different parent while queued. Publishing observations from either replacement could hide worktree changes. Capture the complete directory statx observation and converted stat identity before enumeration. Verify the descriptor mount and recheck both identities afterward. For queued children, resolve the parent through the held child descriptor and compare it with the recorded parent identity. Add the mount identifier to the shared directory identity and register the Linux scan module with Make, CMake, and Meson. The native DEVELOPER=1 boundary build compiles it, while recorded directory changes prevent the completed scan from being accepted. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
Individually anchored descriptors do not establish that the mount namespace or named worktree root stayed unchanged throughout a scan. A mount replacement can invalidate otherwise consistent directory observations. Accept only ext-family and XFS filesystems with complete root statx and mount-identity data. Capture /proc/self/mountinfo before the scan, compare it at completion, and freshly reopen the named worktree root with O_NOFOLLOW to verify its original complete identity. Probe openat2() without requiring it. Register the topology module in all three Linux builds. The native DEVELOPER=1 boundary build compiles it. Missing namespace proof, unsupported filesystems, changed mount tables, or replaced roots reject the result; the retained mount snapshot adds memory and can reject unrelated namespace changes. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
The separately registered Linux metadata, anchored-open, enumeration, directory-validation, and topology modules cannot safely publish a physical scan by themselves. They must share the existing bulk backend lifecycle so every closing check runs before results are accepted. Assemble those modules into the Linux backend and register the shared and platform objects with Make, CMake, and Meson. Retain the existing requirements that core.preloadIndex and core.preloadIndexBulk are enabled and fsmonitor is disabled. Preserve ordinary preload when a required syscall, filesystem, mount proof, or closing validation is unavailable. Cap Linux scans at 16 workers. Each worker can allocate a 1 MiB directory buffer; mount snapshots and retained scan results add further memory. Document ext-family and XFS support and keep directory-type injection confined to the documented test environment. Add and register t7532-preload-index-linux.sh with 12 Linux-only cases. Native Linux validation passes 12/12 in the threaded build and 12/12 in a separate NO_PTHREADS build; CMake and Meson link Git. The suite compares ordinary status for tracked changes, visible and ignored paths, false type hints, fallback shapes, and a synchronized child replacement. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
The bulk preloader rejected an active fsmonitor provider, so status verified ambiguous tracked entries through a separate semantic scan. Publishing bulk observations or refreshed stat data before the closing provider query would permit a concurrent change to invalidate a clean result. Pass held parent descriptors, basenames, and observed metadata from both platform walkers to semantic_verify_file_at(). Borrow the captured provider proof epoch, hash eligible raw-safe files during the bulk walk, and retain clean states and stat updates provisionally. After the closing provider query confirms the same epoch, validate all pending positions before publishing clean states, refreshed stat data, and fsmonitor-valid bits. Clear provisional state on provider failure, epoch mismatch, or invalid updates, and retain the existing complete-refresh fallback. Choose the provider-backed bulk path from its actual safety conditions, not from whether semantic history is awaiting adoption. This lets a trivial daemon response or daemon restart rebuild and close an ordinary bulk proof, including for a skipHash index, while retaining the complete proof epoch and closing query. Require both preload settings, an expanded index, a pending built-in IPC token, and an eligible whole-worktree request. Keep APFS and Linux within their platform and filesystem limits. Allocate a bounded hash buffer and attribute check per content-verification worker, and retain tracked states and stat updates only until closure. Extend the APFS and Linux tests with same-size, restored-mtime content changes. Cover accepted closure, provider failure, dirty status, daemon token reset with a null-checksum index, and Trace2 evidence of hashing, deferred publication, and token acceptance or rejection. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
A live exclude-source proof uses filesystem identity to keep one observation coherent. That identity cannot compare equivalent ignore sources captured by separate status processes: replacing a file with the same contents changes its identity without changing ignore semantics. Hash the existing, validated observations in first-observation order. Frame the digest with its version, source object format, unique source count, path, lookup policy, presence, and content identity. Exclude transient stat identity so an equivalent replacement retains the same semantic digest. Extend the existing exclude-proof unit tests to capture independent proofs across a same-content replacement and repeated observation. The digest is independently testable without issuing a sidecar or changing normal exclude-source validation. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
A later status invocation cannot safely reuse an empty result unless its persistent record identifies the exact index and semantic inputs that the original scan proved. Accepting truncated, ambiguous, or forward-versioned records would turn a cache miss into a false clean result. Define the version-one CSTS encoding and serialize index identity in fixed-width network-byte-order fields. Bind the index format, entry count, checksum, HEAD tree, configuration and repository hashes, one exclude digest, and a bounded builtin-provider token. Protect the complete record with the repository's object-format checksum. Reject unknown flags, unsupported index formats, null required object IDs, invalid token bounds or prefixes, bad checksums, truncation, and trailing payload. Add fixed-width identity and sidecar unit coverage for both SHA-1 and SHA-256. Register the new source and unit suite in both Make and Meson. This patch defines and tests the format; it neither writes a sidecar nor changes status dispatch. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
A valid sidecar encoding is not sufficient if its named index can be replaced between proof capture and publication. Publishing that record would let a later reader associate one clean result with another index. Expose the existing index-snapshot open and named-path revalidation helpers at their first store consumer. Require a durable index identity on local APFS, a matching index format, entry count, and checksum, and agreement between the held descriptor and the named index. Encode the sidecar under its own lockfile and repeat the index checks before committing that lock. Register the store unit suite with Make and Meson. Its local-APFS tests cover successful installation for SHA-1 and SHA-256 and rejection when the source index is replaced after pinning. Other filesystems fail closed. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
A clean provider response does not establish that every index entry can be represented by an empty status result. Conflicted entries, submodules, sparse entries, intent-to-add entries, and independently trusted stat state can all require ordinary index processing. Introduce a single conservative certifiability check. Require a non-null index checksum and provider-valid ordinary entries. Reject gitlinks, nonzero stages, intent-to-add, skip-worktree, CE_VALID, and unrecognized entry flags while allowing the explicitly supported in-memory flags. Extend the existing index unit suite to exercise accepted ordinary entries and each unsupported entry shape. The classifier does not issue a proof or change status behavior by itself. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
An empty status result cannot certify the next invocation unless its tracked and untracked observations, ignore sources, provider token, configuration, repository, HEAD, and named index belong to one completed scan. Publishing a digest before provider-token closure, or reusing cached replacement-ref state, could issue a false clean proof. Retain the standard-exclude digest produced by the complete bulk scan. Keep provider-originated digest state pending until token closure accepts it, and preserve the accepted digest when consuming single-use tracked results. Inspect a fresh, uncached ref store and reject effective replacement refs. Then fingerprint the held local-APFS index and worktree, repository paths, locale, and external attribute state. Issue a sidecar only for the literal, top-level, empty porcelain-v2 command after persistent semantic history, an eligible expanded index, a complete untracked scan, and the HEAD cache tree all agree. Install against the pinned index before rolling back its held index lock; otherwise retain ordinary index-update behavior. Also enable the preceding external-history checkpoint path only for a literal normal status with no pathspec. After the full scan, publish a complete checkpoint for the closed token. A successful save or restore rolls back the acceleration-only index update, preserving another Git implementation's physical index namespace. Optional-lock-free and index-changing commands keep the ordinary path. Add the focused sidecar integration suite and register its source and production code with the relevant Make and Meson builds. Cover prior semantic history, unchanged index contents, exact command shape, rejection of unsupported exact-sidecar inputs, namespace-specific external-history restoration across index re-encoding, and failed checkpoint republication. No early status answer is introduced here. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
An installed sidecar cannot be inspected safely by opening an untrusted adjacent path without bounds. A symbolic link, named pipe, oversized record, or growing file could redirect the read, block status, or consume unbounded memory. Open the named sidecar without following symbolic links and request a nonblocking descriptor. Accept only a regular file of at most 8192 bytes, read exactly its recorded size, reject an additional byte, and parse its checksummed contents into caller-owned storage. Clear failed records and release storage explicitly. Platforms without nonblocking support fail closed. Extend the registered store unit suite to cover owned token storage under both object formats, symbolic links, FIFOs, and an oversized 8193-byte record. The loader is testable at this boundary; it does not yet bypass index deserialization. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
Validating a sidecar must recapture standard excludes before status can trust an empty result. Opening an exclude source that has become a named pipe may otherwise block the supposedly cheap validation. Add an explicit nonblocking flag to exclude-source proof creation and carry it into the existing anchored source-open operation. Reject unknown flags, request nonblocking captures for sidecar issuance, and update the existing bulk-scan and unit-test callers to pass zero, preserving their current blocking and symbolic-link policies. Add a focused FIFO unit test showing that an opted-in proof captures and validates an empty pipe without waiting. The later early-status consumer can reuse nonblocking capture without changing ordinary exclude handling. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
An issued clean-status sidecar has no latency benefit while status still deserializes the index before checking it. Moving the check earlier is safe only if the recorded proof is revalidated around an empty builtin-fsmonitor delta. Attempt the sidecar only for the literal top-level porcelain-v2 command on an eligible main worktree. Load the bounded record, pin the named local-APFS index, recapture excludes without blocking, and check configuration, attributes, repository identity, HEAD, and provider mode. Query the builtin provider directly from the stored token. Keep the attribute and exclude proofs alive across that query. Recheck configuration, HEAD, fresh replacement-ref and repository state, attribute contents and namespace, exclude-source identity, and both the held and named index before accepting an empty delta. Return without deserializing index entries only when every check succeeds; otherwise continue through ordinary status. Unsupported anchored-open platforms take that ordinary path. Register the fast-path source with Make and Meson. Extend the existing sidecar integration suite for read-only hits, dirty worktree shapes, loose, packed, and custom replacement refs, sidecar and exclude FIFOs, changed configuration, attributes, HEAD, null-checksum indexes, and post-query replacement or exclude races. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
A clean-status sidecar is a narrowly scoped proof, not an alternate index or a general cache. Documenting only its serialized bytes would hide the full-scan issuance requirement and the revalidation needed before an empty provider response can answer status. Document the adjacent sidecar path, local-APFS and main-worktree eligibility, fixed-width version-one CSTS fields, a checksum using the repository object-format hash, the separate repository-identity hash, a bounded builtin-provider token, and the 8192-byte read limit. Explain why the source index, configuration, repository, HEAD, attributes, and standard excludes must remain coherent. Describe completed-scan issuance, persistent provider history, held index locks, the post-query race fence, nonblocking source opens, and read-only hits. State that every missing, unsupported, stale, malformed, or raced proof falls back to ordinary status. Register the technical document in both the documentation Makefile and Meson. Also document resumable CSHS history checkpoints separately from CSTS exact-result sidecars. Specify their namespace binding, complete FSMN, UNTR/FSUC, and FSCF contents, canonical logical-index digest, bounded local store, scratch-state validation, publication requirements, and normal-status-only rollback behavior. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
ttaylorr-oai
force-pushed
the
tb/codex/status-preview-unstable
branch
2 times, most recently
from
August 8, 2026 07:26
abc124f to
e106175
Compare
ttaylorr-oai
changed the title
friel-openai
approved these changes
Aug 9, 2026
A normal status after another Git implementation rewrites the index can restore an external clean-history checkpoint, but later invocations still read the index and compute both logical-index digests. On a 1,034,481-entry worktree, that left clean status around 0.5 to 1.0 seconds after the original tracked-file thrash was gone. After a successful external-history restore, let literal top-level plain status publish the same physical clean proof used by the exact porcelain-v2 path. A later hit keeps the normal long-status printer and refreshes branch, tracking, and in-progress-operation state, but skips index deserialization, both logical digests, and untracked traversal. The Apple-written index in this workload uses index.skipHash, so its trailer cannot bind the proof. Accept a zero trailer only when the existing local-APFS durable identity binds the parsed and named index. Rewriting the same logical entries still changes that identity and invalidates the proof. A controlled Apple Git rewrite in the same worktree took 2.58 seconds to re-establish the proof; the next warm plain status took 0.01 seconds and traced clean-proof/hit without do_read_index or history_logical_digest. A new physical rewrite still takes the external-history path once before later unchanged-index runs become fast.
ttaylorr-oai
force-pushed
the
tb/codex/status-preview-unstable
branch
from
August 10, 2026 01:17
e106175 to
039b0c7
Compare
FSUC records each directory validity bit, but valid_recursive is an in-memory summary rebuilt while scanning. External clean-history restore parses UNTR and FSUC into scratch state and pairs their token with FSMN, then hands that cache to status without rebuilding the summary. A root-only fsmonitor event therefore leaves the restored root looking non-recursive, and read_directory walks every cached subtree. Recompute valid_recursive when matching FSMN and FSUC tokens make the cache valid. This only folds already-validated child bits; later invalidation still clears ancestors through the existing path. Extend the FSUC parser test with a root and child so the paired-token transition checks both directory bits and their recursive summary.
5f74c83117 (status: checkpoint clean history outside the index, 2026-08-07) binds each CSH1 payload to a logical digest of every ordered cache entry. A plain status after a foreign index rewrite must compute that digest before restore, then compute it again before reissuing the checkpoint. On the OpenAI checkout those two walks were the bulk of the roughly 900ms touch-x regression. Add a v2 CSH1 source alias for local APFS: durable stat identity, index version, entry count, and trailer checksum. Once repo_read_index() and a pinned snapshot prove the parsed index is that exact physical source, reuse the checkpoint logical hash instead of hashing every entry. Old v1 records remain readable and fall back to the digest. After status, reuse that source hash only when cache_changed contains only FSMN/UNTR acceleration changes, no cache entry changed, all flags remain in the digest accepted set, and sparse checkout is off. Any other state keeps the old digest-and-compare path. This lets an untracked-only status publish its advanced external FSMN/UNTR checkpoint without writing the main index or paying a second digest. Cover v1/v2 parsing and physical snapshot matching in unit tests. Extend the external-history test with a nested cache and a root-only dirty event; it must restore through the alias, avoid both digest regions, visit only the root, and publish updated external history.
ttaylorr-oai
force-pushed
the
tb/codex/status-preview-unstable
branch
from
August 10, 2026 06:29
039b0c7 to
b910009
Compare
The APFS bulk-preload race tests pause status until the test driver writes a byte to a resume FIFO. The child currently publishes its ready file before it opens the FIFO. If it is descheduled between those operations, the parent can observe readiness, write and close its descriptor, and discard the byte before a reader exists. Status then blocks forever in strbuf_read_file(), leaving a macOS CI job apparently hung. Open the resume FIFO first and read from that descriptor after publishing readiness. The parent opens the FIFO read/write before starting status, so the child open cannot block. Readiness now proves a reader is attached, and the resume byte cannot be lost. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
ttaylorr-oai
force-pushed
the
tb/codex/status-preview-unstable
branch
from
August 10, 2026 07:24
b910009 to
597f818
Compare
abg-OAI
approved these changes
Aug 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The APFS bulk-preload test barrier published its READY file before it
opened the resume FIFO. If the child was descheduled at that point, the
parent could observe READY, write and close the FIFO, and lose the byte
before any reader existed. The child then blocked forever while macOS CI
appeared hung.
Open the resume FIFO before publishing READY. The parent already holds
the FIFO open read/write before it starts status, so the child open does
not block; READY now proves that a reader owns the resume descriptor.
The previous topic tip is already published as
v2.55.0-openai.619.gb264c6f26648, so this correction is additive on topof that released history rather than rewriting it. Its resulting tree is
byte-identical to the separately audited owner-amended replay.
Validation:
preload_index_bulk_darwinunit suite (6/6)122/122 ordered parents