Skip to content

Add Git-aware clean mode with grouped candidates - #399

Merged
Byron merged 2 commits into
mainfrom
dua-clean
Sep 12, 2026
Merged

Byron merged 2 commits into
mainfrom
dua-clean

Conversation

@Byron

@Byron Byron commented Sep 9, 2026

Copy link
Copy Markdown
Owner

Tasks

This section is for Byron only. Models continuing this PR must not add, remove, check, uncheck, rename, or reorder checkboxes here.

  • refackiew

Everything below this line was generated by Codex.

Created by Codex on behalf of Byron. Byron will review before this is ready to merge.

Changes

dua clean [DIRECTORY]... finds disposable directories, sizes them, and opens the existing TUI. Multiple sibling candidates share a virtual parent; deeper candidates join their nearest containing group, keeping a Python project's caches together. Group sizes and counts include only the candidates. Marking or deleting a group operates on its candidates and preserves the parent and ordinary source files. Single candidates outside groups remain separate roots.

Discovery reads directory entries/types before detailed size scans and prunes matched artifacts and Git internals. Supported names cover Node dependencies, Python caches and virtual environments, Cargo build output, and Zig's .zig-cache, legacy zig-cache, and zig-out. Cargo and virtualenv candidates require project markers. Within Git worktrees, candidates must be ignored and wholly untracked, including conflict stages and gitlink/sparse ancestors. Failed Git checks, nested repositories, unreadable contents, traversal exclusions, and filesystem boundaries reject the candidate.

Group refresh revalidates recorded members using their original ignore-pattern roots without discovering siblings or scanning the real parent. Navigation is restored and groups that lose their siblings are flattened. Full refresh repeats discovery in the original search directories. Marks are cleared before indices are reused. The command supports --depth, traversal options, --once, and --no-entry-check; snapshot import/export and parent scanning are unavailable.

Discovery submits new candidates to a running walker that shares the existing thread budget. Candidates are sized concurrently and published independently, allowing a fast cache to appear while an earlier, slower cache is still scanning. Workers check new roots between directory jobs; the pool stays alive through discovery gaps and finishes after submissions close and all roots complete. The single-thread path remains sequential, and at most 32 candidates are active.

Each candidate has a separate staging tree. Shared hardlink/APFS accounting is committed only after validation, so rejected candidates cannot consume or undo another candidate's accounting. Closing submissions does not block on a full output channel, and walker shutdown disconnects that channel before joining workers.

Active scans also redraw once a second during gaps in traversal events, keeping counters and visible rows current without keyboard input. Refreshing from inside a candidate revalidates the whole owning candidate and restores the current directory if it remains safe.

Discovery and candidate scans conservatively protect ASCII case variants of Git markers, including .GIT gitfiles pointing outside the search directory. The regression covers lower, upper, and mixed-case spellings. Git reference: 1630431f32, t/t0001-init.sh separate-git-dir cases.

Validation

  • Deterministic regression holds one root blocked while a newly submitted root completes with the same two workers. Further submissions work after the pool becomes idle.
  • Scheduling, full-output cancellation, 40 candidates with 1/2/4 threads, and interleaved hardlink/APFS acceptance/rejection regressions pass.
  • Linux and Windows cross-compilation pass.
  • A regression holds the input and traversal channels quiet and verifies that counters, rows, and the terminal buffer update during the scan. A live terminal scan also produced output in every second without keyboard input.
  • Interior refresh regressions cover both r and R at multiple depths, including a newly added Git marker outside the selected subtree.
  • Regression tests first reproduced missing discovery/publication, unsafe tracked candidates, conflict-stage/gitlink gaps, changing ignore scope on refresh, walker shutdown, grouping containment, and missing Zig cache discovery.
  • Cleanup journeys cover nested grouping, aggregate sizes, partial/full mark toggles, actual deletion with source preservation, similarly prefixed projects, and refresh without widening scope.
  • Zig fixtures cover current and legacy cache names, build output, subtree pruning, ignored caches, unignored directories, and tracked output.
  • make check-pre-push: feature combinations, workspace tests, formatting, Clippy, and stateless journeys.
  • cargo clippy --workspace --all-targets --all-features -- -D warnings.
  • Read-only PTY smoke: Ghostty's .zig-cache and zig-out appeared together
    beneath a virtual parent with two candidates.
  • Snapshot fixtures initialize the new runtime grouping flag; the wire format is unchanged.

Git behavior reference: 1630431f32 (v2.55.GIT), Documentation/git-clean.adoc, and t/t7300-clean.sh cases for ignored tracked directories, nested worktrees, and submodules. Gitoxide's gitoxide-core/src/organize.rs informed repository-boundary pruning. Ghostty's build layout and ignore patterns informed Zig support.

Reported issue

$issue-full-auto Add a `dua clean` subcommand which similar to https://github.com/ByteAtATime/oweka provides a TUI that offers directories to delete. Re-use the existing TUI for `interactive` mode, but list only directories that were found using heuristics as roots.
To find these roots, consider using a search that is shallow, and uses well-known directory names, with an initial traversal that doesn't query metadata, just directory entries and types. Once a root is found, it's scheduled for a detailed traversal to learn about file-sizes as well and ultimately be added as root to the interactive tree. `--export` should naturally not be included in `clean`.

Additional user guidance:

And when Git is compiled in and available, double-check that the candidates aren't actually tracked to avoid proposing the deletion of important directories. Maybe also consider ambiguity, i.e. expect deletable items to be gitignored.

@Byron Byron changed the title Add a Git-aware clean subcommand Add Git-aware clean mode and collapsible TUI panels Sep 9, 2026
@Byron
Byron force-pushed the dua-clean branch 5 times, most recently from 31f1e20 to 4c02a18 Compare September 10, 2026 10:54
@Byron Byron changed the title Add Git-aware clean mode and collapsible TUI panels Add Git-aware clean mode with grouped candidates Sep 10, 2026
@Byron
Byron force-pushed the dua-clean branch 9 times, most recently from d053db3 to fd59c37 Compare September 12, 2026 05:15
Byron pushed a commit that referenced this pull request Sep 12, 2026
Repair the Linux and Windows failures in PR #399's interactive clean journeys.

- Locate the rendered group by its current row instead of assuming that it
  follows the singleton: directory sizes differ across filesystems. Keep
  checking that unrelated rows survive deletion without assuming their order.
- Compare the ordinary browser title with the canonical current directory,
  including Windows' verbatim path prefix.
- Match Git annotation paths against both the discovered worktree spelling
  and its canonical spelling. Git discovery simplifies Windows paths while
  clean candidates retain canonical prefixes; the mismatch hid ignored entries.
  Canonicalize the repository root once, leaving entry paths and symlinks intact.

- Normalize Windows separators only in the Git glob matcher input, so scoped
  candidate names such as `git\__pycache__` match basename patterns.

Validation:

- `make check-pre-push`
- `cargo clippy --offline --workspace --all-targets --all-features -- -D warnings`
- Workspace checks with all targets/features for Windows MSVC and Linux GNU.
- Existing full and partial group-mark assertions and Gitignore checks remain.
Byron and others added 2 commits September 12, 2026 09:29
Accept newly discovered roots in a running fixed-size worker pool. Keep
per-root predicates and completion accounting, and prioritize submitted
roots so existing directory trees cannot starve them.

- Add `RootSender`, `stream_roots()`, and `Walk::next_cancellable()`.
- Disconnect bounded output before joining workers during teardown.
- Preserve native enumeration and bounded parallel metadata processing.
- Cover independent completion, submission priority, input closure,
  full-channel shutdown, and cancellation.

Assisted-by: GPT 6.0
Co-authored-by: GPT 6.0 <codex@openai.com>
This commit was mostly rubber-stamped, but tested for a week or so before that.
By now it's fair to accept that `dua` is in "tool that needs to work, without
needing to be very familiar with how it works" territory, while I still look
at the code at least but without the requirement to understand it all.

<!-- agent -->
Add `dua clean [--depth N] [DIRECTORY]...` to discover disposable build
artifacts and caches, size accepted candidates, and offer them in a dedicated
cleanup hub. Keep the TUI responsive during scanning, deletion, and trashing.

## Discovery and safety

- Search directory entries and types before collecting size metadata. Recognize
  Node dependencies, Python caches and virtual environments, Cargo build output,
  and Zig caches and output. Require project markers for ambiguous names.
- In Git repositories, require expendable ignore matches and reject tracked
  contents, conflict stages, gitlinks, and sparse ancestors. Consult HEAD when
  the index is absent and reject candidates when validation fails.
- Reject nested repositories, case variants of Git markers, unreadable contents,
  excluded descendants, and disallowed filesystem boundaries.
- Feed discovered roots into dua-core's running walker within the configured
  thread budget. Bound active candidates and publish each independently after
  sizing and validation; commit hardlink/APFS accounting only on acceptance.
- Search without a depth limit by default; `--depth` bounds discovery. Support
  traversal filters and `--once`, with no snapshot import/export or parent
  scanning in clean mode.

## Cleanup hub and browsing

- Title the hub with its input directories, group siblings and contained
  candidates, and keep rows in largest-first order as scan results arrive.
- Open groups as scoped views of existing candidate nodes. Reuse ordinary
  sorting, glob search, annotations, navigation, and marking within the group;
  keep group membership and controls local to the hub.
- Group marks target actual candidates. Complete partial marks and toggle fully
  marked groups off, preserving the real parent and unrelated contents.
- Refresh recorded candidates without widening their deletion scope. Revalidate
  whole candidates with their original discovery roots, restore browser/search
  positions, and clear stale marks before tree indices are reused.

## Background deletion

- Run deletion and trash work off the main thread, updating remaining bytes in
  the tree, mark pane, and progress message about once per second.
- Allow navigation, search, sorting, pane controls, and suspend/resume while
  freezing marks and blocking further filesystem actions until the batch ends.
- Preserve bounded filesystem concurrency, parallel file removal, and removal
  of child directories before parents. Batch tree updates using scanned sizes
  without double-counting overlapping marks or completed subtrees.
- Keep failed entries with their remaining sizes and errors. On quit, cancel
  pending work, apply in-flight results, and join workers. On event-loop failure,
  cancel and disconnect events before joining. Trash progress advances when
  each native trash call returns.

Assisted-by: GPT 6.0
Co-authored-by: GPT 6.0 <codex@openai.com>
@Byron
Byron merged commit ed276c2 into main Sep 12, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant