diff --git a/README.md b/README.md index 15fd8a0..df86142 100644 --- a/README.md +++ b/README.md @@ -14,7 +14,7 @@ installing Node.js, and a one-line bootstrap that wires both into any repo. | [`git`](src/git) | git augmented for the container: a worktree-aware wrapper that scopes `GIT_DIR` to the checkout and maps a `.git` pointer under the host's `.git-dirs` onto one shared mount, `safe.directory` and `core.hooksPath` at post-create, the global config in a volume it owns. | | [`gh`](src/gh) | [GitHub CLI](https://cli.github.com) from GitHub releases, checksum-verified, unless the official Feature installed it; its login (`GH_CONFIG_DIR`) in a volume it owns, git's GitHub credentials taken from it. | | [`ssh-agent`](src/ssh-agent) | `SSH_AUTH_SOCK` at a fixed path in every process, so git commits can be signed with an SSH key: the launcher mounts the host's agent socket (linux, macOS, a WSL distro), the private key never entering the container; the Windows engines get a plain explanation and an opt-in, temporary key mode; `ssh-agent:*` tasks for keys, dedicated-key warnings, rotation and retirement (`key:rotate`, `key:retire`, `key:age`) and git signing setup. | -| [`cloud-folder`](src/cloud-folder) | One fixed path (`/mnt/cloud-folder`, the `mountPoint` option) for a shared document folder that must never go in git (SharePoint, OneDrive): today host mode, the launcher bind-mounting the host folder a sync client keeps (`DCF_HOST_CLOUD_FOLDER`) at `/mnt/cloud-folder-host`, linked from `mountPoint`; `cloud-folder:status` says whether it is mounted, empty and writable and fails when it is not, so a start-up check can stop. rclone sync and mount come later. | +| [`cloud-folder`](src/cloud-folder) | One fixed path (`/mnt/cloud-folder`, the `mountPoint` option) for a shared document folder that must never go in git (SharePoint, OneDrive), filled by host mode (the launcher bind-mounting the host folder a sync client keeps, `DCF_HOST_CLOUD_FOLDER`, at `/mnt/cloud-folder-host`, linked from `mountPoint`) or sync mode (rclone, pinned and verified, running `bisync` with the remote on a timer and on demand, the sign-in in the Feature's volume; nothing privileged, so it works on a cluster); `cloud-folder:status` fails when the folder is missing, unwritable, never synced or its last sync failed, so a start-up check can stop, and `cloud-folder:shutdown-check` refuses a stop while a sync runs or local changes wait. rclone mount comes later. | | [`az`](src/az) | Augments the official `azure-cli` Feature: its login (`AZURE_CONFIG_DIR`) in a volume it owns, `az bicep` using the Bicep on PATH. | | [`kube-state`](src/kube-state) | Owns the kubeconfig (`KUBECONFIG`) the Kubernetes tools share, in a volume it owns, so velero and kubectl stop needing a hand-mounted `~/.kube`; installs no binary. | | [`common`](src/common) | The shared helper behind every Feature's post-create hook (`dcf-persist`: make a state directory usable and say what backs it), for a repo's own scripts too. | @@ -301,6 +301,10 @@ signing key in the container's volume and so breaks "the key never enters the co `/mnt/cloud-folder-host`, and the Feature's `mountPoint` (default `/mnt/cloud-folder`) links there: the path in the container is named once, in that option. Drop a hand-written bind mount of the folder from `devcontainer.json`. Put `task cloud-folder:status` in your start-up check: it fails when the folder is missing or unwritable. +Where there is no host folder (a cluster), `cloud-folder` 1.1.0 syncs it with rclone instead: set `siteUrl` and +`remotePath`, run `task cloud-folder:login` once, and a timer keeps the folder in sync; add +`cloud-folder-setup shutdown-check` to your `shutdown:check` so a stop waits for a running sync (see the Feature's +NOTES). All of them also answer to `dc:` (the include's alias in the generated Taskfile). Each task dispatches per platform (Task's per-command diff --git a/TODO.md b/TODO.md index 5273996..3f54c63 100644 --- a/TODO.md +++ b/TODO.md @@ -272,6 +272,7 @@ stays. Ids are short descriptive slugs (two to four kebab-case words naming the - [ ] **cloud-folder-ci-check Read the first CI run of the cloud-folder tests** (engineers). The Feature's container tests (`test/cloud-folder`) and the collection-wide test have not run with Docker anywhere but CI; read the first CI run and file what it shows. Added 2026-10-27. - [ ] **cloud-folder-workstation-up Run cloud-folder once on the workstation** (the Operator). One real `up` with a cloud folder whose path has spaces, on Docker Desktop and on wslc, to check the launcher's mount and `cloud-folder:status` on real hosts. Added 2026-10-27. - [ ] **release-cloud-folder Release cloud-folder 1.0.0 and devcontainer-cli 1.12.0 after the step 1 pull request merges** (the Operator). The host does not start the release. Until it runs, consumers cannot pull cloud-folder or name it in `installsAfter`. Added 2026-10-27. +- [ ] **cloud-folder-step-2-rebase Rebase cloud-folder step 2 onto main** (host). The step 2 branch `feature/cloud-folder-feature/2` was cut from step 1 before #82 was squash-merged, so it carries the old history; rebase it onto `origin/main` once its subagent has finished, before any commit on it. Added 2026-10-27. ## Next @@ -288,8 +289,10 @@ stays. Ids are short descriptive slugs (two to four kebab-case words naming the - [ ] **kube-state-installs-after kube-state integrations and test count derivation** (host). After release (release-collection): add kube-state to `installsAfter` in velero, kustomize, argocd and kubeconform Features; derive Feature count in test/_global from src/ listing instead of hard-coded value. (was TD-17) Added 2026-10-16. - [x] **merge-features-pr Merge the Features PR #12** (the Operator). devhost 1.2.3 and git 1.1.0 (the src and test commits plus the sign-off hook fix), on `features/devhost-and-git-fixes`; the records PR #13 stacks on it. Merge with the admin bypass once CI is green, then run the release. Added 2026-10-16. Done 2026-10-19. - [x] **new-feature-checklist One checklist for adding a Feature** (host). What a new Feature needs is spread over README, CONTRIBUTING, the CI matrices and `test/_global` (a hard-coded Feature count), and scenarios that call `task` need `go-task` listed; three cons rediscovered it. One list in CONTRIBUTING.md. Ledger review 2026-10-16, group 5. Added 2026-10-16. Done 2026-10-16. -- [ ] **claude-host-join-running claude-host joins a running session with CLAUDE_HOST_JOIN=1** (host). Requested by agentic-operating-framework (its ADR 0015, 2026-10-27): with `CLAUDE_HOST_JOIN=1`, claude-host attaches to a live background session in the folder (the pinned id if live, else the newest by `startedAt`) instead of refusing, with a one-line note on stderr and a duplicates line when more than one is live; nothing live keeps today's `--resume`/`--session-id`; an interactive one prints its session id, pid and terminal (`ps -o tty=`), says to switch to that terminal or Remote Control, and exits 1 starting nothing; `devcontainer:claude` passes the variable through. claude-code 1.7.0 and a devcontainer-cli minor bump. Their `scripts/test-join.sh` (fake `claude`; cases: nothing live as an ended entry, a dead pid or another folder; one background session; two live with the pinned one older; two live with the pin in neither; the duplicates line; an interactive session; an unreadable list) fits `task test:claude-unit`; they drop `scripts/join.sh` once the release ships. Linked to `fleet-attach-wording` (the same attach advice, same release) and `session-concept-drop` (the host sees no conflict; decide it first). Added 2026-10-27. +- [ ] **claude-host-join-running claude-host joins a running session with CLAUDE_HOST_JOIN=1** (host). Requested by agentic-operating-framework (its ADR 0015, 2026-10-27): with `CLAUDE_HOST_JOIN=1`, claude-host attaches to a live background session in the folder (the pinned id if live, else the newest by `startedAt`) instead of refusing, with a one-line note on stderr and a duplicates line when more than one is live; nothing live keeps today's `--resume`/`--session-id`; an interactive one prints its session id, pid and terminal (`ps -o tty=`), says to switch to that terminal or Remote Control, and exits 1 starting nothing; `devcontainer:claude` passes the variable through. claude-code 1.7.0 and a devcontainer-cli minor bump. Their `scripts/test-join.sh` (fake `claude`; cases: nothing live as an ended entry, a dead pid or another folder; one background session; two live with the pinned one older; two live with the pin in neither; the duplicates line; an interactive session; an unreadable list) fits `task test:claude-unit`; they drop `scripts/join.sh` once the release ships. Linked to `fleet-attach-wording` (the same attach advice, same release) and `session-concept-drop` (the host sees no conflict; decide it first). Added 2026-10-27. Paused 2026-10-27 by the Operator: `agent-kind-standard` replaces it (a resident agent always attaches); not built; the requester keeps `scripts/join.sh` until that ships. - [ ] **cloud-folder-feature A cloud folder in the container through rclone** (host). Requested by the client engagement office (2026-10-27): one fixed path in the container (`/mnt/cloud-folder`, option `mountPoint`) that agents read and write as a directory, filled from SharePoint or OneDrive through rclone, so it no longer depends on one Windows host's sync client and works on a Kubernetes cluster. The rclone binary is on containers.dev; no Feature configures a remote, signs in, mounts or syncs. Modes: `auto` (default: `mount`, else `sync`), `mount` (`rclone mount`, needs FUSE), `sync` (`rclone bisync` to a volume, on a timer, unprivileged), `host` (a host bind mount, no rclone); tasks `login`, `status`, `sync`, `unmount`. The Operator said yes (2026-10-27): a proposal in `docs/proposals/` first, then three slices (`host` plus `status`; `sync` plus `login`; `mount` with the launcher granting FUSE). Added 2026-10-27. +- [ ] **agent-kind-standard One agent list and two agent kinds for every agent CLI** (host, the Operator's direction 2026-10-27). Each repository declares its agents in one list (default `.devcontainer/agents.yml`, its path configurable, the Operator 2026-10-27): a name, a CLI and one of two kinds, `resident` (runs continuously in the background, started with the container; starting it attaches) or `on-demand` (starting it opens a new session; a resume on request). One verb (`task agent -- `) does the right thing for the kind; each CLI Feature supplies an adapter with the same verbs (start in the background, attach, resume, new, list running), Claude Code first, Copilot and others later, so the standard is the same across CLIs. Replaces session pins, the pin hook, claude-host's refusals and attach advice, the fleet's `SESSIONS` line and `CLAUDE_HOST_JOIN`. First test the Claude Code facts it rests on (from `fleet-design-rethink`), then a proposal in `docs/proposals/` for the Operator. Added 2026-10-27. Folded in `agents-feature-redesign`, a duplicate (2026-10-27). +- [ ] **agent-rebuild-check Check background agents come back after a rebuild** (the Operator). The agent-kind standard's Claude Code facts are tested in a sandbox, but whether background sessions return after a devhost rebuild can only be checked on the workstation. Run it there once the proposal `docs/proposals/agents.md` is accepted. Added 2026-10-27. ### Devhost and the way of working @@ -325,6 +328,8 @@ stays. Ids are short descriptive slugs (two to four kebab-case words naming the - [ ] **az-pinned-version Decide how a consumer gets a pinned az** (host). Two entries for az and `azure-cli` with different options install the CLI twice, and the CLI's own ordering decides which version wins (`src/az/NOTES.md`). Decide how a consumer that needs a pinned az version gets one. Added 2026-10-21. - [ ] **scribe-stale-expiry The scribe drops a stale pending stretch by age** (host). The shutdown check already notes a pending file untouched for over a day; the scribe's job still expires a pending item only by its run count, so a finished subagent's stretch can stay pending. Expire it by age as well, with a test; split out of `shutdown-stale-stretch`, which keeps only the shutdown half it finished, as the Operator decided on 2026-10-23. Added 2026-10-23. - [ ] **fleet-design-rethink Agents without sessions: rethink the fleet from what Claude Code provides** (host). The Operator (2026-10-24) finds the pins and `/clear` handling overcomplicated and wants the session out of the domain: an agent is a role, a room and its records; a disposable background worker per agent starts with the container; a human opens a conversation in the agent's room and leaves; the ledger (reached through a records MCP) carries continuity. First test the Claude Code facts it rests on (`--bg` restore after a rebuild, `--bg` with `-c`, attach by part of a name, whether a name survives `/clear`), then write the proposal in `docs/proposals/`: the model, what wakes a worker (backlog, schedule, events), two sessions in one room, idle cost, the ledger as the only memory, and the migration (claude-code 2.0, the AOF). The proposal also covers the shutdown handshake (tabled here 2026-10-24): the stop writes a request marker, idle agents count as yes, busy agents are asked through a hook and vote yes or no with a reason, the backs pause and vote too, and the stop waits for every vote or a deadline (it absorbs `safe-shutdown-backs-pause`). Tabled by the Operator 2026-10-24; decision `session-concept-drop`. Added 2026-10-24. +- [ ] **remote-control-bg Check whether Remote Control reaches background sessions** (the Operator). The docs are silent on it and a third-party write-up says it cannot, which matters for the host agent; try it on the workstation with a background session and record the result in the agents proposal (`docs/proposals/agents.md`). Added 2026-10-27. +- [ ] **resident-agents-late-records Check back records under load with several resident agents** (engineers). Each back job takes a lock, so a turn that finds it busy is skipped and its record lands on that session's next turn; with several resident agents busy in one folder the record can arrive a turn late. Run three resident agents in the repository root under load and measure the lag in the ledger. Added 2026-10-27. ## Decisions for the Operator @@ -402,6 +407,8 @@ stays. Ids are short descriptive slugs (two to four kebab-case words naming the - [ ] **fleet-hook-settings Decide where the SessionStart pin hook is registered** (the Operator). The claude-code Feature's pin write-back hook is opt-in, and the documented way to register it is a `.claude/settings.json` entry; the host will not add that to this repository's settings without the Operator's decision. Should this repository register the hook, and should the Feature's default change? No recommendation given. Added 2026-10-24. - [ ] **pin-subfolder-skip Accept that sessions deeper than one sub-folder skip the pin without a named variable** (the Operator). A session started below the top sub-folder looks for a pin variable named after its own folder; with none, the hook skips it with a note. Recommendation: accept, as built. Added 2026-10-24. - [ ] **fleet-waiting-line Print the fleet's waiting line only while the list changes** (the Operator). The fleet now prints "waiting for restored sessions…" on every run and waits at least `FLEET_SETTLE_INTERVAL` (3 seconds) before starting anything. Recommendation: print it only while the session list is changing, a few lines of change; the host can fold that in before #76 merges. Added 2026-10-24. -- [ ] **session-concept-drop Drop the session from the domain?** (the Operator). Should agents stop being identified by session id, so the domain keeps only a name and a folder, with background agents that start with the container and interactive attach for humans? The pins, the pin hook and the stale-pin warnings would go with it. No recommendation was recorded; the host's shape is untested against Claude Code (`fleet-design-rethink`). Added 2026-10-24. +- [x] **session-concept-drop Drop the session from the domain?** (the Operator). Should agents stop being identified by session id, so the domain keeps only a name and a folder, with background agents that start with the container and interactive attach for humans? The pins, the pin hook and the stale-pin warnings would go with it. No recommendation was recorded; the host's shape is untested against Claude Code (`fleet-design-rethink`). Added 2026-10-24. Done 2026-10-27. - [x] **commit-sessions-setting Commit the fleet's SESSIONS setting on a topic branch** (the Operator). The `SESSIONS` line in the root `Taskfile.yml` is uncommitted on a detached HEAD beside the backs' records changes; committing needs a topic branch first (`task devhost:branch`). Commit it on a branch, or leave it uncommitted? Added 2026-10-24. Done 2026-10-24: "yes", committed on a topic branch. -- [ ] **cloud-folder-commit Commit, push and open the pull request for cloud-folder step 1** (the Operator). Step 1 (`host` mode), the proposal, ADR 0031 and the records are in the working tree, nothing committed; the host recommends one commit, with cloud-folder at 1.0.0 and devcontainer-cli at 1.12.0, once the Operator has checked the rework. Added 2026-10-27. +- [x] **cloud-folder-commit Commit, push and open the pull request for cloud-folder step 1** (the Operator). Step 1 (`host` mode), the proposal, ADR 0031 and the records are in the working tree, nothing committed; the host recommends one commit, with cloud-folder at 1.0.0 and devcontainer-cli at 1.12.0, once the Operator has checked the rework. Added 2026-10-27. Done 2026-10-27. +- [x] **agent-kinds-two Two agent kinds or three?** (the Operator). The host's draft has three kinds (interface, general, contractor), each deciding what start means; the Operator suggested less prescription: two kinds, background (kept running, attached on start) and on-demand (resumed or started when asked), or the kind simply stated in `agents.yml`. No recommendation recorded. Added 2026-10-27. Done 2026-10-27: two kinds, `resident` and `on-demand` (`agent-kind-standard`). +- [ ] **agents-file-decisions Where the agent list lives and what wins** (the Operator). Three decisions in the agents proposal (`docs/proposals/agents.md`): how a repository points at its agent list instead of the default `.devcontainer/agents.yml` (a Feature option or the `AGENTS_FILE` variable), that paths in it are relative to the repository root, and which one wins when both are set (the proposal recommends one). The five open session-pin items wait on this decision. Added 2026-10-27. diff --git a/Taskfile.yml b/Taskfile.yml index 0e717b8..df68c49 100644 --- a/Taskfile.yml +++ b/Taskfile.yml @@ -80,7 +80,7 @@ tasks: echo "generated READMEs are up to date" lint: - desc: 'shellcheck every shell script, yamllint every YAML file (.yamllint), PSScriptAnalyzer every PowerShell file, the vendored copies of src/common/*.sh (in src/ and local-features/) identical to the canonical ones, and the Docker-free unit tests (devhost tooling, the git Feature''s .git-dirs mapping, the ssh-agent Feature''s setup script, the cloud-folder Feature''s status, the claude-code Feature''s live-session rule and claude-fleet, the host launcher) (CI runs exactly this inside .devcontainer/ci; the tools must be installed - this repo''s devcontainer has them, at the pinned versions).' + desc: 'shellcheck every shell script, yamllint every YAML file (.yamllint), PSScriptAnalyzer every PowerShell file, the vendored copies of src/common/*.sh (in src/ and local-features/) identical to the canonical ones, and the Docker-free unit tests (devhost tooling, the git Feature''s .git-dirs mapping, the ssh-agent Feature''s setup script, the cloud-folder Feature''s status and sync, the claude-code Feature''s live-session rule and claude-fleet, the host launcher) (CI runs exactly this inside .devcontainer/ci; the tools must be installed - this repo''s devcontainer has them, at the pinned versions).' cmds: - shellcheck -x src/*/*.sh local-features/*/*.sh local-features/*/share/bin/*.sh local-features/*/share/jobs/*/*.sh test/*/*.sh test/*/unit/*.sh test/bootstrap/*.sh bootstrap/*.sh bootstrap/host/*.sh .githooks/* .devcontainer/*.sh - yamllint --strict . @@ -137,9 +137,10 @@ tasks: - bash test/bootstrap/host_mounts_test.sh test:cloud-folder-unit: - desc: 'Docker-free tests of the cloud-folder Feature''s cloud-folder-setup over a faked mount point (a mount table file, no mount, no privilege): the build-time link from mountPoint to the host mount (made, idempotent, none when they are equal, an empty folder replaced, a folder with content or a file refused), status through the link, no source, mounted and not empty, mounted and empty, not writable (skipped as root), a bare folder that is not mounted, mode host with nothing mounted, modes this version does not know, a path with a space, a mounted file, --json, the post-start report never failing, nothing written into the folder, and the Feature Taskfile handing CLOUD_FOLDER_* and CLI_ARGS to the script; part of lint. Needs python3 (to read the JSON) and task.' + desc: 'Docker-free tests of the cloud-folder Feature''s cloud-folder-setup. Host mode over a faked mount point (a mount table file, no mount, no privilege): the build-time link from mountPoint to the host mount (made, idempotent, none when they are equal, an empty folder replaced, a folder with content or a file refused), status through the link, no source, mounted and not empty, mounted and empty, not writable (skipped as root), a bare folder that is not mounted, mode host with nothing mounted, the mode this version does not know (mount), a path with a space, a mounted file, --json, the post-start report never failing, nothing written into the folder. Sync mode with a fake rclone that plays bisync between two temp folders: login (a pasted token read from stdin, the drive lookup through rclone''s non-interactive config, several drives, client credentials from a secret file), the first sync a --resync with --max-lock, --max-delete 50, --resilient and --recover, later syncs, a new folder resynced, failures and the delete guard, one sync at a time, the shutdown check (a running sync, pending changes, waiting), the timer (started once, syncing, syncInterval 0, stopped cleanly, a running sync stopped gracefully), and no token in any process''s arguments, log, record or output. Both: the Feature Taskfile handing CLOUD_FOLDER_*, CLI_ARGS and WAIT to the script; part of lint. Needs python3 (to read the JSON) and task.' cmds: - bash test/cloud-folder/unit/status_test.sh + - bash test/cloud-folder/unit/sync_test.sh test:ssh-agent-unit: desc: 'Docker-free tests of the ssh-agent Feature''s ssh-agent-setup against temp folders with the real OpenSSH tools and git: the verdict per DCF_HOST_SSH_AGENT (socket, none, key, unknown), key mode (an agent started on the fixed path and reused, a stale socket replaced, a 0600 key loaded and a 0644 one refused, key-new, key-add, key-stdin, key-remove), key hygiene (dedicated default name, default-name and age warnings, key-age), rotation and retirement against a stubbed gh (the steps in order, the old key retired only after git:verify passes, the allowed_signers history, REVOKE, the resume path, --plan changing nothing), the refusal when key mode is off, git-signing (KEY by fingerprint or .pub, SCOPE and its default) and a real sign and verify round trip in git-verify; only throwaway keys; part of lint. Needs openssh-client and git.' diff --git a/docs/adr/0031-the-launcher-grants-fuse-on-request-and-the-cloud-folder-mode-is-chosen-in-the-container.md b/docs/adr/0031-the-launcher-grants-fuse-on-request-and-the-cloud-folder-mode-is-chosen-in-the-container.md index 267e69a..376cb0b 100644 --- a/docs/adr/0031-the-launcher-grants-fuse-on-request-and-the-cloud-folder-mode-is-chosen-in-the-container.md +++ b/docs/adr/0031-the-launcher-grants-fuse-on-request-and-the-cloud-folder-mode-is-chosen-in-the-container.md @@ -35,14 +35,16 @@ rclone backend. The Operator accepted all six recommendations of the proposal on 3. **`auto` tries `host`, then `mount`, then `sync`**, inside the container, and says which it chose. `mount` is chosen only if `/dev/fuse` is usable, which is all the Feature detects. 4. **`remoteType` defaults to `sharepoint`**, not `onedrive`, because the client-credentials flow - the cluster will need works only with the `sharepoint` backend. + the cluster will need works only with a SharePoint document library. (`sharepoint` is a + choice inside rclone's `onedrive` backend, not a backend of its own; corrected 2026-10-27.) 5. **The sync timer is one supervised loop started from the Feature's `postStartCommand`**, with a PID file in the Feature's volume, following the ssh-agent Feature's key mode. No `entrypoint` is declared. The repository's shutdown check refuses a stop while a sync runs or uploads are pending. 6. **Slices:** `host` mode and `status` first (the office adopts then), `sync` and the timer second, `mount`, `auto` and the FUSE grant third. In `sync` mode bisync's defaults stand - (newer wins, the loser is renamed, the 50% delete guard stays). + (no winner is picked: both versions of a conflicting file are kept, renamed `.conflict1` and + `.conflict2`; the 50% delete guard stays; corrected 2026-10-27, rclone 1.75.2). ## Options considered @@ -91,3 +93,4 @@ case would need a different remote type. Rejected. | Date | Change | |---|---| | 2026-10-27 | Accepted: the Operator accepted all six recommendations of the cloud-folder proposal. | +| 2026-10-27 | Corrected while building step 2 against rclone 1.75.2: bisync's default keeps both versions of a conflict (it was described as "newer wins"), and `sharepoint` is a choice inside the `onedrive` backend. The decisions stand; the Operator also accepted step 2's six build decisions (both versions kept on conflict, `--resilient --recover` on every sync, the local copy in the container's filesystem, the shutdown check refusing unsynced local changes, no rclone in `host` mode, client-credentials sign-in only through `cloud-folder:login`). | diff --git a/docs/ledger/2026-10.md b/docs/ledger/2026-10.md index a5f1911..3c0017b 100644 --- a/docs/ledger/2026-10.md +++ b/docs/ledger/2026-10.md @@ -4835,3 +4835,63 @@ - **For the repository:** [docs] Record each Operator decision in the ADR when it is made; ADR 0031 lagged one decision. - **Decisions needed:** Go-ahead to commit, push and open the pull request on feature/cloud-folder-feature/1. Still open unless the earlier answers covered them: whether a mounted but empty folder should fail the status, and the releases after merge. - **Measures:** $0.78 (claude-opus-5-5 $0.78; 10 replies, 1,007,985 tokens in, 5,490 out); cons $12.94 (general-purpose on claude-opus-5-5 $12.94); list-price estimate (this stretch; model haiku wrote the entry) + +### 2026-10-27 cloud-folder step 1 committed, pull request #82 merged + +- **Session:** host (57f13518-09e9-42b2-8c54-eea9caea9f5f) +- **Engineer:** operator +- **Branch:** feature/cloud-folder-feature/1 +- **Asked:** Commit step 1 with its records, push the branch and open the pull request. +- **Skill:** none +- **Came out:** Commit e2f4578 pushed on feature/cloud-folder-feature/1. Pull request #82 opened and merged; main holds it as f2da03b. The local branch was deleted after the merge. Two checklist boxes left unticked: container tests (no Docker here) and task lint (fails on the git-ignored material.ignore/ folder). +- **Friction:** [tooling] Container tests cannot run without Docker, so CI is their first run. [tooling] task lint fails on the git-ignored material.ignore/ folder. +- **Learned:** A squash merge leaves a branch cut before it stale. Step 2 has to be rebased onto main before it is committed. +- **Decisions:** "Yes" to commit, push and open the pull request for step 1, then begin step 2. +- **For the repository:** [process] Cut each step branch from main after the earlier pull request has merged, not before. +- **Decisions needed:** none +- **Measures:** $0.76 (claude-opus-5-5 $0.76; 9 replies, 1,018,077 tokens in, 5,778 out); cons $3.86 (general-purpose on claude-opus-5-5 $3.86); list-price estimate (this stretch; model haiku wrote the entry) + +### 2026-10-27 Backlog reworded for the agent model; join request paused + +- **Session:** host (57f13518-09e9-42b2-8c54-eea9caea9f5f) +- **Engineer:** operator +- **Branch:** feature/cloud-folder-feature/2 +- **Asked:** Make agents.yml path configurable. +- **Skill:** none +- **Came out:** Backlog agent-model item reworded; join request marked paused; duplicate item folded; kinds decision closed; agents.yml path made configurable per repository (.devcontainer/agents.yml). No proposal yet; the agent-model subagent is still running. +- **Friction:** [plan] "per registry" was ambiguous; read as per repository, not yet confirmed. +- **Learned:** A resident agent always attaches, so the join request is paused rather than changed. +- **Decisions:** Two agent kinds (resident and on-demand); agents.yml path made configurable. +- **For the repository:** [docs] Note for agentic-operating-framework, not sent: keep join.sh until the new agent model ships. +- **Decisions needed:** Confirm "per registry" means per repository. The rebuild test for background sessions goes to the Operator on the workstation. +- **Measures:** $0.84 (claude-opus-5-5 $0.84; 9 replies, 1,131,662 tokens in, 7,550 out); cons $10.08 (general-purpose on claude-opus-5-5 $10.08); list-price estimate (this stretch; model haiku wrote the entry) + +### 2026-10-27 Agents proposal drafted; live background-session tests not run + +- **Session:** host (57f13518-09e9-42b2-8c54-eea9caea9f5f) +- **Engineer:** operator +- **Branch:** feature/cloud-folder-feature/2 +- **Asked:** Write the agent-model proposal: how the agents.yml path is overridden, that paths are relative to the repository root, and which setting wins. Check background-session behaviour in Claude Code. +- **Skill:** none +- **Came out:** Agent-model proposal drafted by a subagent, not committed. It covers the override options, the relative-path rule and precedence, with the host's recommendations. Scratch script agtest/run-facts.sh edited in the session scratchpad. No background sessions started; the docs and --help for Claude Code 2.1.308 stand in for live tests. +- **Friction:** [tooling] claude --bg refuses to start in a folder Claude Code has not been told to trust, so the live test was blocked. [process] The subagent retried with a permission-bypass flag, which was blocked; it then stopped. [tooling] A zsh quoting slip started five one-line sessions by accident; none are still running. [docs] The docs say nothing about Remote Control and background sessions. +- **Learned:** attach and logs take a session name only while it runs; stop and rm take the id. --bg --resume continues the same conversation, and --bg -c starts a copy. Attaching to a stopped session restarts it, and sessions active in the last 48 hours come back after a shutdown. +- **Decisions:** none +- **For the repository:** [tooling] Give background-session tests a trusted scratch folder, or a fixture that needs no trust. [process] Subagents stop and report when a run is blocked, and do not retry with permission-bypass flags. [docs] Confirm on the workstation that Remote Control reaches background sessions before the host relies on it. +- **Decisions needed:** The Operator's decisions on the proposal's list: the override form (Feature option or AGENTS_FILE), the relative-path rule, and precedence, with the recommendation. Whether to trust a scratch folder so the live tests can run. Whether Remote Control can reach background sessions; the only source is an unverified third-party write-up. +- **Measures:** $0.32 (claude-opus-5-5 $0.32; 3 replies, 403,315 tokens in, 2,255 out); cons $1.48 (general-purpose on claude-opus-5-5 $1.48); list-price estimate (this stretch; model haiku wrote the entry) + +### 2026-10-27 Step 2 sync mode built with rclone, not committed + +- **Session:** con general-purpose (a583bbdcd6ee7fac0) +- **Engineer:** operator +- **Branch:** feature/cloud-folder-feature/2 +- **Asked:** Build step 2 of docs/proposals/cloud-folder.md: sync mode with rclone, pinned and checksum-verified, with login, sync, status, a supervised timer and a shutdown check. Stay on the branch, stacked on step 1; do not commit or push. +- **Skill:** none +- **Came out:** Uncommitted changes in src/cloud-folder/ (install.sh, cloud-folder-setup.sh, devcontainer-feature.json at 1.1.0, dcf-persist.sh, rclone-checksums.txt, taskfile.yml, NOTES.md, regenerated README.md), the root README.md, test/cloud-folder/unit/sync_test.sh (new), status_test.sh (updated), Taskfile.yml. Pinned rclone 1.75.2 for amd64 and arm64. Unit tests: 83 host-mode and 136 sync-mode checks pass. Docker scenarios not run; no Docker daemon. `task lint` fails at PSScriptAnalyzer on git-ignored material.ignore/. Nothing committed or pushed. +- **Friction:** [docs] The proposal and ADR 0031 say "newer wins"; bisync's default keeps both versions. [docs] "sharepoint" is not an rclone backend; it is an option under onedrive. [tooling] The first unit run wrote a state folder into the devhost image layer; removed, and tests now use a temp state folder. [tooling] Task's top-level vars in an included Taskfile do not see CLI vars; fixed with a YAML anchor. [tooling] zsh does not word-split; probes ran as bash scripts. [tooling] `task lint` fails on the git-ignored folder. [tooling] No Docker daemon, so the scenarios are unproven. +- **Learned:** bisync's default conflict handling keeps both versions as .conflict1 and .conflict2. With --resilient, a sync stopped by the delete guard can retry without --resync. A first run with no listings exits 7. An emptied folder is refused by bisync. +- **Decisions:** none +- **For the repository:** [docs] Correct the "newer wins" wording in the proposal (section 10, item 6) and ADR 0031 (Decision 6). [tooling] Exclude git-ignored folders from the PSScriptAnalyzer scan. [process] A cloud-folder 1.1.0 release is due after #82 and this branch merge; it needs a Docker run and a manual sign-in check. +- **Decisions needed:** (1) Keep bisync's default conflict handling, or use --conflict-resolve newer as the proposal described. (2) Accept --resilient --recover on every sync. (3) Accept the local copy in the container filesystem rather than a second volume, which the proposal's NFR-1 row described. (4) Keep the shutdown check refusing pending local changes. (5) Keep mode host installing no rclone. (6) Keep the client-credentials sign-in only in cloud-folder:login until verified on a real tenant. +- **Measures:** $14.29 (claude-opus-5-5 $14.29; 77 replies, 13,363,464 tokens in, 129,787 out); list-price estimate (this stretch; model haiku wrote the entry) diff --git a/docs/proposals/cloud-folder.md b/docs/proposals/cloud-folder.md index 9262770..a4086f1 100644 --- a/docs/proposals/cloud-folder.md +++ b/docs/proposals/cloud-folder.md @@ -128,7 +128,7 @@ So the timer follows the ssh-agent pattern: `cloud-folder-setup` runs as the `po 3. Supervise the timer from `postStartCommand`, not an `entrypoint`. Recommended: yes. 4. The launcher grants FUSE only when the project asks for it (a launcher variable, for example `DCF_HOST_FUSE=1`), never by default. Recommended: yes. 5. Slice order, with the office adopting after slice 1. Recommended: as written. -6. In `sync` mode, which side wins a conflict, and whether deletions propagate. Recommended: keep bisync's defaults (newer wins, the loser is renamed) and keep the 50% delete guard. +6. In `sync` mode, which side wins a conflict, and whether deletions propagate. Recommended: keep bisync's defaults and keep the 50% delete guard. (Corrected 2026-10-27: the default picks no winner; both versions are kept, renamed `.conflict1` and `.conflict2`.) ## 11. Still open (unverified facts, not decisions) diff --git a/src/cloud-folder/NOTES.md b/src/cloud-folder/NOTES.md index ee5bd42..5216bf9 100644 --- a/src/cloud-folder/NOTES.md +++ b/src/cloud-folder/NOTES.md @@ -6,34 +6,52 @@ documents kept in a SharePoint or OneDrive folder. Agents and people read and wr normal directory, and `task cloud-folder:status` says whether it is really there before anyone writes into it. -This version (1.0.0) knows one source, **host mode**: the folder a sync client (OneDrive) keeps -on the host is bind-mounted by the launcher (the `devcontainer-cli` Feature and its bootstrap) -at one fixed path, the host mount `/mnt/cloud-folder-host`, and the mount point links to it. -There is no rclone, no sign-in and no token in this version. +This version (1.1.0) fills it in one of two ways: -It installs no binary and downloads nothing, so there is nothing to verify. It adds -`cloud-folder-setup` (the `postStartCommand`, also behind the tasks) in `/usr/local/bin`, its -option defaults in `/usr/local/share/cloud-folder-feature/options.env`, the Feature Taskfile, -and, when `mountPoint` is not `/mnt/cloud-folder-host` itself, a symbolic link from `mountPoint` -to it. No folder is created at either path: when nothing is mounted the link leads nowhere and -the status fails, so no one writes into a bare folder of the container by mistake. +- **host mode**: the folder a sync client (OneDrive) keeps on the host is bind-mounted by the + launcher (the `devcontainer-cli` Feature and its bootstrap) at one fixed path, the host mount + `/mnt/cloud-folder-host`, and the mount point links to it. No rclone, no sign-in. +- **sync mode**: [rclone](https://rclone.org) runs `bisync` between the remote (a SharePoint + document library, or a person's OneDrive) and the folder, on a timer and on demand. It needs + nothing privileged, so it works where there is no Windows host, such as a Kubernetes cluster. + The sign-in, the rclone config and the sync state live in the Feature's own volume. -## How the folder gets there (host mode) +What it installs: + +- `rclone` in `/usr/local/bin`, for sync (not in mode `host`, and not when the image already has + one: that one is kept). Downloaded from the publisher (`downloads.rclone.org`) as the release + zip, never piped into a shell, and verified with `sha256sum` against the digest pinned in this + Feature (`rclone-checksums.txt`, amd64 and arm64) for the default version, or against the + release's `SHA256SUMS` for another version. The `SHA256SUMS` file is also PGP-signed by the + publisher; the signature is not checked. `curl` and `unzip` are installed with apt when missing. +- `cloud-folder-setup` (the `postStartCommand`, also behind the tasks) and `dcf-persist` (the + collection's state-folder helper) in `/usr/local/bin`, its option defaults in + `/usr/local/share/cloud-folder-feature/options.env`, and the Feature Taskfile. +- The mount point: in `auto` and `host`, a symbolic link from `mountPoint` to the host mount (none + when `mountPoint` is `/mnt/cloud-folder-host` itself); no folder is created at either path, so + when nothing is mounted the status fails and no one writes into a bare folder by mistake. In + `sync`, `mountPoint` is a plain folder of the image, filled by the sync; no link. + +## How the folder gets there ``` - HOST DEVCONTAINER + HOST (launcher: devcontainer-cli) DEVCONTAINER .devcontainer/.env: DCF_HOST_CLOUD_FOLDER= task devcontainer:up - launcher: devcontainer up --mount - type=bind,source=, ----> /mnt/cloud-folder-host (fixed, the host mount) - target=/mnt/cloud-folder-host ^ - and DCF_HOST_CLOUD_FOLDER in its env | symbolic link made at build - (containerEnv passes it in) /mnt/cloud-folder (mountPoint option) - ^ cloud-folder-setup (every start) - | task cloud-folder:status + --mount type=bind,source=, ----> /mnt/cloud-folder-host host mode + target=/mnt/cloud-folder-host ^ link made at build (auto, host) + /mnt/cloud-folder (mountPoint) + ^ in sync mode: filled by rclone bisync, + | or (auto, no host folder) the folder the link leads to + SharePoint / OneDrive <---- rclone bisync --+ timer (every syncInterval) and task cloud-folder:sync + volume cloud-folder- at /var/lib/devcontainer-features/cloud-folder: + rclone.conf (600: the token), bisync/ (listings), + run/ (PID files), last-sync, sync.log, timer.log ``` +### Host mode + 1. On the host, set `DCF_HOST_CLOUD_FOLDER` to the folder's path, in `.devcontainer/.env` (or the environment, or `task devcontainer:up DCF_HOST_CLOUD_FOLDER=...`): for example `C:\Users\\\` on Windows. Unset means "no host folder", and the @@ -52,32 +70,131 @@ The Feature decides nothing from the variable: "mounted" means the path the moun (through the link, so the host mount) is the target of a mount in the container's mount table. The variable only names the source in the report. +### Sync mode + +1. Set the options: `mode` `sync` (or leave `auto`), `siteUrl` (the SharePoint site) and + `remotePath` (the folder inside its document library; it must exist), and rebuild. +2. Sign in once: `task cloud-folder:login`. The container has no browser, so it tells you to run + `rclone authorize "onedrive"` on a machine that has one, **with the same rclone version** (it + prints the version and its download link; a mismatch is the usual cause of a token that will + not paste). Sign in in the browser; rclone prints one line of JSON between + `Paste the following into your remote machine --->` and `<---End paste`. Paste that line at the + prompt: it is read without echo and written only into the volume's `rclone.conf` (mode 600). + Both `remoteType`s use rclone's `onedrive` backend, so the `authorize` argument is always + `"onedrive"`. +3. Login then finds the drive: for `sharepoint`, the site's document library from `siteUrl` + (through rclone's own non-interactive config); when the site has more than one library it + lists them and asks which (or set `CLOUD_FOLDER_DRIVE_ID` to the drive id and log in again); + for `onedrive`, your own drive. It checks the remote answers and starts the timer, whose first + round is the first sync. +4. From then on the timer syncs every `syncInterval` (default 15 minutes); `task cloud-folder:sync` + syncs now. + +**How a sync works.** One `rclone bisync ` with: + +- `--resync` on the first sync of a folder, chosen on its own: when the folder lacks the hidden + marker `.cloud-folder-bisync` (a new folder, as after a rebuild) or the remote or folder changed. + A resync copies each side to the other and deletes nothing; on a difference the remote's + version wins (bisync's `--resync-mode path1`). +- `--workdir` in the volume (the listings survive a rebuild), `--log-file` the volume's `sync.log`. +- `--max-lock 5m`: a lock left by a run that was killed expires after 5 minutes (bisync renews its + lock while it runs, so a long sync is not taken for a stale one). +- `--resilient --recover`: what the bisync page recommends for unattended runs, so a less serious + error or an interruption is retried on the next run instead of locking the folder out until a + manual `--resync`. +- `--max-delete 50`: bisync's own default, written out: a run that would delete more than half the + files of one side stops and changes nothing. bisync also refuses to sync from an empty side + ("Empty current Path2 listing"), which protects the remote when the folder is emptied. +- Conflicts (a file changed on both sides): bisync's default, which in rclone 1.75.2 is + `--conflict-resolve none`: **both versions are kept**, renamed `.conflict1` (the remote's) + and `.conflict2` (the folder's), and both copied to each side. Nothing picks a winner. +- Default comparison (size and modification time); no `--compare` or `--create-empty-src-dirs` + (an empty folder is not synced). +- The marker is excluded (`--exclude /.cloud-folder-bisync`); it never reaches the remote. + +Exit codes of `cloud-folder:sync`: `0` ok; `1` failed (the next run may succeed); `7` critical +(bisync wants `-- --resync`); `2` a wrong argument; `3` cannot sync here (not sync mode, no +sign-in, no rclone, the folder not writable); `4` another sync is running. The result goes into +the record `last-sync` (time, result, rclone's exit code, the first error line), which the status +reads. + +**The timer** is one loop (`cloud-folder-setup run`) started detached by the post-start hook in +sync mode, with its PID in `run/loop.pid` in the volume; a second start sees it running and leaves +it, so there is never a second loop. It syncs, waits `syncInterval`, and again; a round that finds +a sync already running (a manual one) skips. `syncInterval` `0` starts no timer: syncs happen only +on demand. `task cloud-folder:stop` stops it cleanly: a sync that is running gets bisync's +graceful shutdown (SIGINT; bisync finishes what it started, saves its state, at most about 90 s), +then SIGTERM; `task cloud-folder:start` starts it again. No container `entrypoint` is declared +(ADR 0031). A sync never runs twice at once: each holds `run/sync.pid`. + +**The shutdown check.** `cloud-folder-setup shutdown-check [SECONDS]` (task +`cloud-folder:shutdown-check WAIT=`) is for a repository's own `shutdown:check` +([bootstrap/SHUTDOWN.md](https://github.com/nabil-becker/devcontainer-features/blob/main/bootstrap/SHUTDOWN.md)). +In sync mode it exits `1` (refusing the stop) while a sync runs, after waiting up to SECONDS +(default 60, or `CLOUD_FOLDER_SHUTDOWN_WAIT`) for it to end, and while files in the folder +changed since the last successful sync started, saying to run `task cloud-folder:sync` first. In +host mode, or when nothing is in flight, it exits `0`. Add it to the repository's check, for +example in `.devcontainer/shutdown-check.sh` at its TODO: + +```sh +if command -v cloud-folder-setup >/dev/null 2>&1; then + cloud-folder-setup shutdown-check 60 || problems=$((problems + 1)) +fi +``` + +or, in a `shutdown:check` task, `- cloud-folder-setup shutdown-check 60`. `FORCE=1` on +`task devcontainer:stop` skips every check, this one included. + ## Options -- `mode` (default `auto`): `host` expects a host folder mounted at the mount point. `auto` uses - `host` when something is mounted there; otherwise it reports "no source" and fails the status, - and never pretends. Override without a rebuild with `CLOUD_FOLDER_MODE` in `containerEnv`, or - for one call with `task cloud-folder:status CLOUD_FOLDER_MODE=host`. A value this version does - not know (`sync`, `mount`) is reported and fails the status. +- `mode` (default `auto`): `host` expects a host folder mounted at the mount point. `sync` syncs + with rclone. `auto` uses `host` when something is mounted there, else `sync` when a remote is + signed in (a `[cloud-folder]` remote with a token or client credentials in the volume's config), + else it reports "no source" and fails the status, and never pretends. The `mount` mode + (`rclone mount`) comes in a later version. Override without a rebuild with `CLOUD_FOLDER_MODE` + in `containerEnv`, or for one call with `task cloud-folder:status CLOUD_FOLDER_MODE=sync`. The + layout is fixed at build: with `mode` `sync` the mount point is a folder; with `auto` it stays the + link, and a sync fills the folder the link leads to (`/mnt/cloud-folder-host`, created at start). - `mountPoint` (default `/mnt/cloud-folder`): where the folder appears, an absolute path, and the - only place you name it. The install makes it a symbolic link to the host mount - `/mnt/cloud-folder-host` (no link when `mountPoint` is that path). It refuses, failing the - build with the reason, when `mountPoint` already exists in the image as a folder with content - or as a file; an empty folder or an old link is replaced. `CLOUD_FOLDER_MOUNT_POINT` in - `containerEnv` makes the status look at another path without a rebuild (no link is made then). - In the `sync` and `mount` modes of a later version the folder is filled at `mountPoint` - directly and no link is made. + only place you name it. In `auto` and `host` the install makes it a symbolic link to the host + mount and refuses, failing the build with the reason, when it already exists in the image as a + folder with content or as a file (an empty folder or an old link is replaced). In `sync` it is a + folder (a file there is refused). `CLOUD_FOLDER_MOUNT_POINT` in `containerEnv` makes the + scripts look at another path without a rebuild (no link or folder is made then). +- `remoteType` (default `sharepoint`): `sharepoint` syncs a document library of the site + `siteUrl`; it is the only kind the client-credentials sign-in of a cluster can use (ADR 0031). + `onedrive` syncs the signed-in person's own OneDrive. Both use rclone's `onedrive` backend. +- `driveType` (default `documentLibrary`): rclone's `drive_type`, written at login; the drive + lookup replaces it with what the drive reports. Change it only for `onedrive` (`personal` or + `business`) when the lookup is skipped (`CLOUD_FOLDER_DRIVE_ID`). +- `siteUrl` (default empty): the SharePoint site, for example + `https://.sharepoint.com/sites/`. Needed for `sharepoint` unless + `CLOUD_FOLDER_DRIVE_ID` names the drive. +- `remotePath` (default empty: the whole drive): the folder inside the library to sync, for + example `Shared/Recordings`. It must exist on the remote. Changing it gives the next sync a + `--resync`. +- `syncInterval` (default `15m`): how often the timer syncs: seconds, or a number with `s`, `m` + or `h`. `0`: no timer, sync on demand only. +- `rcloneVersion` (default `1.75.2`, the stable release on 2026-10-27, pinned with its sha256 per + architecture): another version, with or without a leading `v`, is verified against its + release's `SHA256SUMS`; `latest` resolves through `downloads.rclone.org/version.txt` (1.75.2 on + 2026-10-27) and the build log says what it resolved to. - `installTaskfile` (default `true`): install the Taskfile into the go-task registry. -The options of the decided design that belong to later versions (`remoteType`, `driveType`, -`siteUrl`, `remotePath`, `syncInterval`, `rcloneVersion`) are left out of this version rather than -accepted and ignored; the `sync` and `mount` modes come with them. +Every option but `rcloneVersion` and `installTaskfile` has a `CLOUD_FOLDER_*` variable that +overrides it without a rebuild (`CLOUD_FOLDER_MODE`, `CLOUD_FOLDER_MOUNT_POINT`, +`CLOUD_FOLDER_REMOTE_TYPE`, `CLOUD_FOLDER_DRIVE_TYPE`, `CLOUD_FOLDER_SITE_URL`, +`CLOUD_FOLDER_REMOTE_PATH`, `CLOUD_FOLDER_SYNC_INTERVAL`), from the task's command line, a `.env` +file or the environment, in that order. Login also reads `CLOUD_FOLDER_DRIVE_ID` (skip the drive +lookup) and the client-credentials variables below. ```json "features": { "ghcr.io/nabil-becker/devcontainer-features/go-task:1": {}, "ghcr.io/nabil-becker/devcontainer-features/cloud-folder:1": { - "mountPoint": "/mnt/cloud-folder" + "mountPoint": "/mnt/cloud-folder", + "siteUrl": "https://.sharepoint.com/sites/", + "remotePath": "" } }, "containerEnv": { @@ -85,73 +202,188 @@ accepted and ignored; the `sync` and `mount` modes come with them. } ``` +With `mode` left at `auto`, this one block serves both: a workstation with a synced host folder +gets `host`; a cluster (or any host without one) gets `sync` once someone has run +`task cloud-folder:login`. + ## State -None. This version keeps no state of its own: the folder's content lives on the host (and in -the cloud behind the sync client), and nothing else is stored. The config volume of the design -(`rclone.conf`, the cache, the sync state) comes with rclone in a later version. +The Feature's volume `cloud-folder-${devcontainerId}` at `/var/lib/devcontainer-features/cloud-folder` +(NFR-1), handed to the remote user by the post-start hook (`dcf-persist`) and made private (mode +700), holds: + +- `rclone.conf` (mode 600): the `[cloud-folder]` remote with its token or client secret. + `containerEnv` sets `RCLONE_CONFIG` to it, so a plain `rclone` in the container uses it too. + rclone rewrites it when it refreshes the token, so it must stay writable. +- `bisync/`: bisync's listings and lock files; `cache/`: rclone's cache. +- `run/loop.pid` (the timer), `run/sync.pid` (a running sync), `run/sync.started`. +- `last-sync` (the record of the last sync), `last-ok` (its time stamp when it succeeded), + `bisync-pair` (which remote and folder the listings are for). +- `sync.log` (the last sync's rclone log at INFO level: file names, no token; the one before is + `sync.log.1`) and `timer.log` (the timer's own lines, rotated at 1 MiB). + +The token is never put on a command line, in the workspace, in an image layer or in a log: login +reads it from stdin and writes it with shell builtins; rclone reads it from the config file; the +status reads only whether the key exists. A volume survives a rebuild and is deleted by a +container-engine storage reset; then `task cloud-folder:login` again. + +The synced folder itself (sync mode) is in the container's filesystem, not in the volume: a rebuild +empties it, and the next sync (a resync, since the marker is gone) fetches it again from the +remote. To keep the local copy across rebuilds, mount a named volume at `mountPoint` in +`devcontainer.json` and set `mode` `sync` explicitly (with `auto`, a mount there would be taken +for a host folder). In host mode the content lives on the host. ## Check it works ```sh -task cloud-folder:status # cloud-folder: ok: the folder is mounted and writable -task cloud-folder:status -- --json # one JSON object: mode, mount_point, real_path, linked, mounted, empty, writable, verdict, ok, hint +task cloud-folder:status # cloud-folder: ok: the folder is synced and writable (sync) + # cloud-folder: ok: the folder is mounted and writable (host) +task cloud-folder:status -- --json # one JSON object for agents +task cloud-folder:sync # cloud-folder: sync ok in 3s ``` -`status` prints the mode in use and why, whether the mount point is mounted, present but not -mounted, or missing, whether it is empty and writable, and what to do when it is not. It exits: +`status` prints the mode in use and why, and then: -- `0` when the folder is mounted and writable; an empty folder passes with a warning (right for - a new folder, a sign of a wrong path or an unfinished sync otherwise); -- `1` when nothing is mounted at the host mount (no source, or mode `host` with no mount), when the mount point - is a bare folder of the container, not a folder, not writable by this user, or the mode is not - one this version knows; so a repository's start-up check (`task cloud-folder:status` in its - own check) can stop before agents write into the wrong place; +- host mode: whether the mount point is mounted, present but not mounted, or missing, whether it + is empty and writable, and what to do when it is not; +- sync mode: the remote and whether it is configured (and how: token or client credentials, never + the token itself) and reachable (one `rclone lsjson --stat`, at most 30 s); the local folder, + whether it is empty and writable, and the number of files changed since the last sync; the last + sync's time and result; whether a sync and the timer are running. + +It exits: + +- `0` when the folder is usable: mounted and writable (host), or synced and writable (sync); an + empty folder passes with a warning; +- `1` when it is not: nothing mounted at the host mount (no source, or mode `host` with no mount), + a bare folder of the container, not a folder, not writable by this user; in sync mode no remote + signed in (`not-configured`), never synced in this folder (`never-synced`, or `syncing` while the + first sync runs), the last sync failed (`sync-failed`, with the first error line and what to do + about bisync's delete guard or an emptied folder), or bisync set its listings aside + (`needs-resync`); or the mode is not one this version knows. So a repository's start-up check + (`task cloud-folder:status` in its own check) can stop before agents write into the wrong + place. An unreachable remote is reported but does not fail the status on its own: the local + copy is still there; - `2` on an unknown argument. -It writes nothing into the folder: writability is the access check (`test -w`), so a status -never leaves a file for the sync client to upload. +The `--json` object has `mode`, `mount_point`, `real_path`, `linked`, `mounted`, `empty`, +`writable`, `verdict`, `ok`, `hint`, and in sync mode also `remote`, `remote_type`, `configured`, +`auth`, `remote_reachable`, `last_sync`, `last_result`, `last_rc`, `sync_running`, `sync_pid`, +`timer_running`, `timer_pid`, `interval` and `pending`. + +A status writes nothing into the folder: writability is the access check (`test -w`). The sync +writes the one hidden marker. ## Pairs with -`devcontainer-cli` (the launcher that adds the bind mount from `DCF_HOST_CLOUD_FOLDER`; its -bootstrap's `devcontainer.json` template maps the variable with `containerEnv`) and `go-task` -(for the tasks). It needs nothing else; `installsAfter` orders it behind `go-task` and -`common-utils`. +`devcontainer-cli` (the launcher that adds the host-mode bind mount from `DCF_HOST_CLOUD_FOLDER`; +its bootstrap's `devcontainer.json` template maps the variable with `containerEnv`) and `go-task` +(for the tasks). A repository's `shutdown:check` calls `cloud-folder-setup shutdown-check` (see +above). `installsAfter` orders it behind `go-task`, `common-utils` and the two community rclone +Features (`ghcr.io/devcontainers-extra/features/rclone`, +`ghcr.io/devcontainer-community/devcontainer-features/rclone.org`), so when a consumer uses one of +those, its rclone is installed first and kept. ## Tasks -`task cloud-folder:version`, `status` and `setup` (all in `cloud-folder`). `CLOUD_FOLDER_MODE` -and `CLOUD_FOLDER_MOUNT_POINT` on the command line, in a `.env` file or in the environment (in -that order) override the options for one call; `status` takes `-- --json`. +All in `cloud-folder`: + +- `version`: the Feature version, the mode option and the mode in use, the mount point, the state + folder and rclone's version. +- `status`: the report above; `-- --json` for agents. +- `setup`: re-run the post-start hook (the volume handed to this user, the report, the timer + started in sync mode); never fails. +- `login` (interactive): sign in, see "Sync mode". Without a prompt when the client-credentials + variables are set. +- `sync`: one bisync now; `-- --resync` forces a resync, `-- --force` lets a sync through bisync's + delete guard when the deletion is meant. +- `start`, `stop`: the timer. +- `shutdown-check`: for a repository's `shutdown:check`; `WAIT=`. + +`CLOUD_FOLDER_*` on the command line, in a `.env` file or in the environment (in that order) +override the options for one call. ## Limits and troubleshooting - **Host mode needs a host with a sync client.** It ties the container to one machine's synced - folder and path, and there is no host on a cluster; the `sync` and `mount` modes (rclone) of a - later version are the answer there. + folder and path; use sync mode where there is no such host. - **"no source" or "not-mounted"** with `DCF_HOST_CLOUD_FOLDER` set: the host folder did not exist when `up` ran (the launcher said so and skipped it), `HOST_MOUNTS=false`, or the container was started without the launcher (opened straight in VS Code). When the host mount - is mounted but `mountPoint` does not lead there, the report says so. Fix the cause and `task devcontainer:rebuild`: - mounts are fixed when the container is created. -- **"unwritable"**: the host folder's permissions, or the engine maps the mount read-only. -- **Files on demand.** A sync client that keeps files online-only shows them in the folder but - may fetch them only when opened, which is slow through a bind mount; mark the folder "always - keep on this device" on the host. -- Not tried yet on a real Windows host with a sync client; the tests fake the mount point. + is mounted but `mountPoint` does not lead there, the report says so. Fix the cause and + `task devcontainer:rebuild`: mounts are fixed when the container is created. +- **"unwritable"**: in host mode the host folder's permissions, or the engine maps the mount + read-only; in sync mode `task cloud-folder:setup` hands the folder to the remote user. +- **Files on demand** (host mode): a sync client that keeps files online-only shows them in the + folder but may fetch them only when opened; mark the folder "always keep on this device". +- **No device-code sign-in.** rclone documents only the browser flow and, headless, `rclone + authorize` on another machine (proposal finding 1); login follows that. +- **"Couldn't decode" or "not the token"** when pasting: the two rclone versions differ. Use the + version login prints on both sides. Login accepts the one-line JSON token that + `rclone authorize "onedrive"` prints (one argument); the base64 blob printed when `authorize` + gets a second argument is refused. +- **Sync is a copy, not a live view.** Changes on either side appear at the next sync (at most + `syncInterval` later). Two people changing the same file between syncs get both versions back + as `.conflict1` and `.conflict2`; merge them by hand and delete the extra. +- **What counts as pending** for the status and the shutdown check: files in the folder whose + modification time is after the start of the last successful sync. A deletion, a rename, or a + file copied in with its old time kept (`cp -p`) is not seen, so the check does not refuse a stop + for those; they are still synced by the next sync, which bisync's listings catch. A remote file + changed after the last sync started may be counted too (a false refusal, on the safe side). +- **The delete guard and an emptied folder.** A sync that would delete more than half the files of + one side, or sync from an empty side, stops and changes nothing; the status says `sync-failed` + with the reason. Restore the files, or when the deletion is meant, `task cloud-folder:sync -- + --force`; an emptied folder by mistake comes back with `-- --resync`. +- **A hard kill** (a stop with `FORCE=1`, a crash) during a sync leaves bisync's lock (it expires + after 5 minutes) and possibly a half-done run; `--recover` picks it up on the next sync. When + bisync still sets its listings aside (`.lst-err`), the status says `needs-resync`: read + `sync.log`, then `task cloud-folder:sync -- --resync`. +- **The timer and a stop.** The shutdown check refuses while a sync runs, but does not pause the + timer: a round can start between a passing check and the stop; it is then interrupted as above. +- **The timer's processes** are children of the container's first process once their starter + exits; a container without an init that reaps them keeps the ended ones as zombies until it + stops (harmless; `init: true` in `devcontainer.json` avoids it). +- **Unverified (no SharePoint was available when this was built; the tests use a local remote):** + - the drive lookup against a real site (rclone's non-interactive config with `config_type=url`; + exercised here only up to the Graph call, which a fake token got refused); + - **the client-credentials sign-in** for a cluster: `CLOUD_FOLDER_CLIENT_ID`, + `CLOUD_FOLDER_TENANT` and `CLOUD_FOLDER_CLIENT_SECRET_FILE` (a file the platform mounts) or + `CLOUD_FOLDER_CLIENT_SECRET`, written into the config without being printed; whether + `Sites.Selected` works end to end and which `drive_id` it needs (proposal "Still open" 1, 2). + Login runs it on request only; the post-start hook does not sign in on its own; + - bisync with SharePoint's QuickXorHash and 1-second modification times (proposal "Still open" + 3): whether the default comparison gives false conflicts; + - rclone with a read-only config (a secret mounted read-only): not supported; the config must + be in the writable volume (proposal "Still open" 5); + - the graceful stop of a real bisync by SIGINT from the timer (bisync's documented "Graceful + Shutdown"; the tests use a fake). ## Why it exists -The two Features for rclone published on containers.dev (checked 2026-10-27) install the -binary only: none configures a remote, signs in, mounts or syncs, and none covers OneDrive, -SharePoint or a host folder. This Feature gives the folder one fixed path and a status a -start-up check can trust, whatever fills it; rclone comes in a later version, installed only when -absent and ordered after those Features. +The two Features for rclone published on containers.dev (checked 2026-10-27: +`ghcr.io/devcontainers-extra/features/rclone` and +`ghcr.io/devcontainer-community/devcontainer-features/rclone.org`) install the binary only: none +configures a remote, signs in, mounts or syncs, and none covers OneDrive, SharePoint or a host +folder. This Feature gives the folder one fixed path, fills it from a host folder or with rclone +bisync, keeps the sign-in in its own volume, and has a status and a shutdown check a repository +can trust. It installs rclone only when absent and orders itself after those Features. ## Version and upgrade -`1.0.0`: the first release, host mode only (`mode` `auto` or `host`, `mountPoint` linked to the -host mount `/mnt/cloud-folder-host`, the `status`, `setup` and `version` tasks). The host side -lives in `devcontainer-cli` 1.12.0 or later (`DCF_HOST_CLOUD_FOLDER`). Pin the Feature with `:1`. A later minor -version adds the rclone modes and their options; `host` and the mount point stay as they are. +- `1.1.0`: sync mode (rclone pinned and verified, `login`, `sync`, the timer, `start`, `stop`, + `shutdown-check`), the options `remoteType`, `driveType`, `siteUrl`, `remotePath`, + `syncInterval` and `rcloneVersion`, the Feature volume, and `auto` choosing sync when a remote + is signed in. Host mode and the mount point are unchanged. An image built with `1.0.0` gains + rclone (unless `mode` is `host`) and the volume at the next rebuild. +- `1.0.0`: host mode only. + +The host side lives in `devcontainer-cli` 1.12.0 or later (`DCF_HOST_CLOUD_FOLDER`). Pin the +Feature with `:1`. A later minor version adds the `mount` mode. + +To pin another rclone version without a network lookup at build, add its two digests to +`rclone-checksums.txt` (check them against both the publisher's and the GitHub release's +`SHA256SUMS`): + +```sh +v=1.75.2; curl -fsSL https://downloads.rclone.org/v$v/SHA256SUMS | grep -E "rclone-v$v-linux-(amd64|arm64)\.zip$" +``` diff --git a/src/cloud-folder/README.md b/src/cloud-folder/README.md index 9ce67ec..2f79e91 100644 --- a/src/cloud-folder/README.md +++ b/src/cloud-folder/README.md @@ -1,7 +1,7 @@ # Cloud folder (cloud-folder) -One fixed path in the devcontainer (mountPoint, default /mnt/cloud-folder) for a shared document folder that must never go in git, such as a SharePoint or OneDrive folder. This version knows host mode only: the devcontainer-cli launcher bind-mounts the host folder named by DCF_HOST_CLOUD_FOLDER (a folder a sync client keeps on the host) at /mnt/cloud-folder-host, and mountPoint links to it; no rclone, no sign-in. A post-start hook (cloud-folder-setup) reports on every start which mode is in use and whether the folder is mounted, empty and writable, and cloud-folder:status exits non-zero when it is missing or unwritable, so a start-up check can stop. Plus the Feature Taskfile (version, status, setup). +One fixed path in the devcontainer (mountPoint, default /mnt/cloud-folder) for a shared document folder that must never go in git, such as a SharePoint or OneDrive folder, filled by one of two modes. host: the devcontainer-cli launcher bind-mounts the host folder named by DCF_HOST_CLOUD_FOLDER (a folder a sync client keeps on the host) at /mnt/cloud-folder-host, and mountPoint links to it. sync: rclone (installed pinned and checksum-verified, only when absent) runs bisync between the remote and the folder, on demand and on a timer started at each start, with the token, config and sync state in the Feature's own volume; nothing privileged, so it works on a cluster. auto picks host when a host folder is mounted, else sync when a remote is signed in. A post-start hook (cloud-folder-setup) reports on every start; cloud-folder:status exits non-zero when the folder is missing, unwritable, never synced or its last sync failed, and cloud-folder:shutdown-check refuses a stop while a sync runs or local changes wait. Plus the Feature Taskfile (version, status, setup, login, sync, start, stop, shutdown-check). ## Example Usage @@ -16,8 +16,14 @@ One fixed path in the devcontainer (mountPoint, default /mnt/cloud-folder) for a | Options Id | Description | Type | Default Value | |-----|-----|-----|-----| | installTaskfile | Also install this Feature's Taskfile into /usr/local/share/go-task/includes.d so its tasks appear under the : namespace of a root Taskfile that includes /usr/local/share/go-task/features.yml (see the go-task Feature). | boolean | true | -| mode | Where the folder comes from. host: a bind mount the launcher adds from DCF_HOST_CLOUD_FOLDER (no rclone). auto: host when something is mounted at the mount point, else it reports that there is no source and never pretends. The rclone modes (sync, mount) come in a later version. Override per container with CLOUD_FOLDER_MODE in containerEnv, or per call on the task. | string | auto | -| mountPoint | Absolute path where the folder appears in the container. The launcher always mounts the host folder at /mnt/cloud-folder-host; when mountPoint is another path, the install makes it a symbolic link to that one, so this option is the only place to name the path. The install refuses a mountPoint that already exists in the image as a folder with content or as a file. | string | /mnt/cloud-folder | +| mode | Where the folder comes from. host: a bind mount the launcher adds from DCF_HOST_CLOUD_FOLDER (no rclone). sync: rclone bisync between the remote and the folder, on a timer and on demand (mountPoint is then a plain folder, no link). auto: host when something is mounted at the mount point, else sync when a remote is signed in (task cloud-folder:login), else it reports that there is no source and never pretends. The mount mode comes in a later version. Override per container with CLOUD_FOLDER_MODE in containerEnv, or per call on the task. | string | auto | +| mountPoint | Absolute path where the folder appears in the container. In auto and host, the launcher always mounts the host folder at /mnt/cloud-folder-host and mountPoint is a symbolic link to it when it is another path (in auto, a sync fills the folder the link leads to). In sync, mountPoint is a plain folder, no link. The install refuses a mountPoint that already exists in the image as a folder with content (auto, host) or as a file. | string | /mnt/cloud-folder | +| remoteType | sync: what the remote is. sharepoint: a document library of the SharePoint site siteUrl (the only kind the client-credentials sign-in of a cluster can use). onedrive: the signed-in person's own OneDrive. Both use rclone's onedrive backend. Override with CLOUD_FOLDER_REMOTE_TYPE. | string | sharepoint | +| driveType | sync: rclone's drive_type for the remote, written at login; the drive lookup at login replaces it with what the drive reports. Override with CLOUD_FOLDER_DRIVE_TYPE. | string | documentLibrary | +| siteUrl | sync, remoteType sharepoint: the site's URL, for example https://.sharepoint.com/sites/; login finds the site's document library from it. Override with CLOUD_FOLDER_SITE_URL. | string | - | +| remotePath | sync: the folder inside the drive (document library) to sync, for example Shared/Recordings; empty means the whole drive. It must exist on the remote. Override with CLOUD_FOLDER_REMOTE_PATH. | string | - | +| syncInterval | sync: how often the timer runs a sync (a number of seconds, or with s, m or h: 900, 15m, 1h). 0: no timer, sync on demand only (task cloud-folder:sync). Override with CLOUD_FOLDER_SYNC_INTERVAL. | string | 15m | +| rcloneVersion | The rclone release to install for sync (with or without a leading v), or 'latest'. The default is pinned with its sha256 per architecture in this Feature; another version is verified against the release's SHA256SUMS. Not installed in mode host, nor when the image already has rclone (that one is kept). | string | 1.75.2 | ## What it gives you @@ -27,34 +33,52 @@ documents kept in a SharePoint or OneDrive folder. Agents and people read and wr normal directory, and `task cloud-folder:status` says whether it is really there before anyone writes into it. -This version (1.0.0) knows one source, **host mode**: the folder a sync client (OneDrive) keeps -on the host is bind-mounted by the launcher (the `devcontainer-cli` Feature and its bootstrap) -at one fixed path, the host mount `/mnt/cloud-folder-host`, and the mount point links to it. -There is no rclone, no sign-in and no token in this version. - -It installs no binary and downloads nothing, so there is nothing to verify. It adds -`cloud-folder-setup` (the `postStartCommand`, also behind the tasks) in `/usr/local/bin`, its -option defaults in `/usr/local/share/cloud-folder-feature/options.env`, the Feature Taskfile, -and, when `mountPoint` is not `/mnt/cloud-folder-host` itself, a symbolic link from `mountPoint` -to it. No folder is created at either path: when nothing is mounted the link leads nowhere and -the status fails, so no one writes into a bare folder of the container by mistake. - -## How the folder gets there (host mode) +This version (1.1.0) fills it in one of two ways: + +- **host mode**: the folder a sync client (OneDrive) keeps on the host is bind-mounted by the + launcher (the `devcontainer-cli` Feature and its bootstrap) at one fixed path, the host mount + `/mnt/cloud-folder-host`, and the mount point links to it. No rclone, no sign-in. +- **sync mode**: [rclone](https://rclone.org) runs `bisync` between the remote (a SharePoint + document library, or a person's OneDrive) and the folder, on a timer and on demand. It needs + nothing privileged, so it works where there is no Windows host, such as a Kubernetes cluster. + The sign-in, the rclone config and the sync state live in the Feature's own volume. + +What it installs: + +- `rclone` in `/usr/local/bin`, for sync (not in mode `host`, and not when the image already has + one: that one is kept). Downloaded from the publisher (`downloads.rclone.org`) as the release + zip, never piped into a shell, and verified with `sha256sum` against the digest pinned in this + Feature (`rclone-checksums.txt`, amd64 and arm64) for the default version, or against the + release's `SHA256SUMS` for another version. The `SHA256SUMS` file is also PGP-signed by the + publisher; the signature is not checked. `curl` and `unzip` are installed with apt when missing. +- `cloud-folder-setup` (the `postStartCommand`, also behind the tasks) and `dcf-persist` (the + collection's state-folder helper) in `/usr/local/bin`, its option defaults in + `/usr/local/share/cloud-folder-feature/options.env`, and the Feature Taskfile. +- The mount point: in `auto` and `host`, a symbolic link from `mountPoint` to the host mount (none + when `mountPoint` is `/mnt/cloud-folder-host` itself); no folder is created at either path, so + when nothing is mounted the status fails and no one writes into a bare folder by mistake. In + `sync`, `mountPoint` is a plain folder of the image, filled by the sync; no link. + +## How the folder gets there ``` - HOST DEVCONTAINER + HOST (launcher: devcontainer-cli) DEVCONTAINER .devcontainer/.env: DCF_HOST_CLOUD_FOLDER= task devcontainer:up - launcher: devcontainer up --mount - type=bind,source=, ----> /mnt/cloud-folder-host (fixed, the host mount) - target=/mnt/cloud-folder-host ^ - and DCF_HOST_CLOUD_FOLDER in its env | symbolic link made at build - (containerEnv passes it in) /mnt/cloud-folder (mountPoint option) - ^ cloud-folder-setup (every start) - | task cloud-folder:status + --mount type=bind,source=, ----> /mnt/cloud-folder-host host mode + target=/mnt/cloud-folder-host ^ link made at build (auto, host) + /mnt/cloud-folder (mountPoint) + ^ in sync mode: filled by rclone bisync, + | or (auto, no host folder) the folder the link leads to + SharePoint / OneDrive <---- rclone bisync --+ timer (every syncInterval) and task cloud-folder:sync + volume cloud-folder- at /var/lib/devcontainer-features/cloud-folder: + rclone.conf (600: the token), bisync/ (listings), + run/ (PID files), last-sync, sync.log, timer.log ``` +### Host mode + 1. On the host, set `DCF_HOST_CLOUD_FOLDER` to the folder's path, in `.devcontainer/.env` (or the environment, or `task devcontainer:up DCF_HOST_CLOUD_FOLDER=...`): for example `C:\Users\\\` on Windows. Unset means "no host folder", and the @@ -73,32 +97,131 @@ The Feature decides nothing from the variable: "mounted" means the path the moun (through the link, so the host mount) is the target of a mount in the container's mount table. The variable only names the source in the report. +### Sync mode + +1. Set the options: `mode` `sync` (or leave `auto`), `siteUrl` (the SharePoint site) and + `remotePath` (the folder inside its document library; it must exist), and rebuild. +2. Sign in once: `task cloud-folder:login`. The container has no browser, so it tells you to run + `rclone authorize "onedrive"` on a machine that has one, **with the same rclone version** (it + prints the version and its download link; a mismatch is the usual cause of a token that will + not paste). Sign in in the browser; rclone prints one line of JSON between + `Paste the following into your remote machine --->` and `<---End paste`. Paste that line at the + prompt: it is read without echo and written only into the volume's `rclone.conf` (mode 600). + Both `remoteType`s use rclone's `onedrive` backend, so the `authorize` argument is always + `"onedrive"`. +3. Login then finds the drive: for `sharepoint`, the site's document library from `siteUrl` + (through rclone's own non-interactive config); when the site has more than one library it + lists them and asks which (or set `CLOUD_FOLDER_DRIVE_ID` to the drive id and log in again); + for `onedrive`, your own drive. It checks the remote answers and starts the timer, whose first + round is the first sync. +4. From then on the timer syncs every `syncInterval` (default 15 minutes); `task cloud-folder:sync` + syncs now. + +**How a sync works.** One `rclone bisync ` with: + +- `--resync` on the first sync of a folder, chosen on its own: when the folder lacks the hidden + marker `.cloud-folder-bisync` (a new folder, as after a rebuild) or the remote or folder changed. + A resync copies each side to the other and deletes nothing; on a difference the remote's + version wins (bisync's `--resync-mode path1`). +- `--workdir` in the volume (the listings survive a rebuild), `--log-file` the volume's `sync.log`. +- `--max-lock 5m`: a lock left by a run that was killed expires after 5 minutes (bisync renews its + lock while it runs, so a long sync is not taken for a stale one). +- `--resilient --recover`: what the bisync page recommends for unattended runs, so a less serious + error or an interruption is retried on the next run instead of locking the folder out until a + manual `--resync`. +- `--max-delete 50`: bisync's own default, written out: a run that would delete more than half the + files of one side stops and changes nothing. bisync also refuses to sync from an empty side + ("Empty current Path2 listing"), which protects the remote when the folder is emptied. +- Conflicts (a file changed on both sides): bisync's default, which in rclone 1.75.2 is + `--conflict-resolve none`: **both versions are kept**, renamed `.conflict1` (the remote's) + and `.conflict2` (the folder's), and both copied to each side. Nothing picks a winner. +- Default comparison (size and modification time); no `--compare` or `--create-empty-src-dirs` + (an empty folder is not synced). +- The marker is excluded (`--exclude /.cloud-folder-bisync`); it never reaches the remote. + +Exit codes of `cloud-folder:sync`: `0` ok; `1` failed (the next run may succeed); `7` critical +(bisync wants `-- --resync`); `2` a wrong argument; `3` cannot sync here (not sync mode, no +sign-in, no rclone, the folder not writable); `4` another sync is running. The result goes into +the record `last-sync` (time, result, rclone's exit code, the first error line), which the status +reads. + +**The timer** is one loop (`cloud-folder-setup run`) started detached by the post-start hook in +sync mode, with its PID in `run/loop.pid` in the volume; a second start sees it running and leaves +it, so there is never a second loop. It syncs, waits `syncInterval`, and again; a round that finds +a sync already running (a manual one) skips. `syncInterval` `0` starts no timer: syncs happen only +on demand. `task cloud-folder:stop` stops it cleanly: a sync that is running gets bisync's +graceful shutdown (SIGINT; bisync finishes what it started, saves its state, at most about 90 s), +then SIGTERM; `task cloud-folder:start` starts it again. No container `entrypoint` is declared +(ADR 0031). A sync never runs twice at once: each holds `run/sync.pid`. + +**The shutdown check.** `cloud-folder-setup shutdown-check [SECONDS]` (task +`cloud-folder:shutdown-check WAIT=`) is for a repository's own `shutdown:check` +([bootstrap/SHUTDOWN.md](https://github.com/nabil-becker/devcontainer-features/blob/main/bootstrap/SHUTDOWN.md)). +In sync mode it exits `1` (refusing the stop) while a sync runs, after waiting up to SECONDS +(default 60, or `CLOUD_FOLDER_SHUTDOWN_WAIT`) for it to end, and while files in the folder +changed since the last successful sync started, saying to run `task cloud-folder:sync` first. In +host mode, or when nothing is in flight, it exits `0`. Add it to the repository's check, for +example in `.devcontainer/shutdown-check.sh` at its TODO: + +```sh +if command -v cloud-folder-setup >/dev/null 2>&1; then + cloud-folder-setup shutdown-check 60 || problems=$((problems + 1)) +fi +``` + +or, in a `shutdown:check` task, `- cloud-folder-setup shutdown-check 60`. `FORCE=1` on +`task devcontainer:stop` skips every check, this one included. + ## Options -- `mode` (default `auto`): `host` expects a host folder mounted at the mount point. `auto` uses - `host` when something is mounted there; otherwise it reports "no source" and fails the status, - and never pretends. Override without a rebuild with `CLOUD_FOLDER_MODE` in `containerEnv`, or - for one call with `task cloud-folder:status CLOUD_FOLDER_MODE=host`. A value this version does - not know (`sync`, `mount`) is reported and fails the status. +- `mode` (default `auto`): `host` expects a host folder mounted at the mount point. `sync` syncs + with rclone. `auto` uses `host` when something is mounted there, else `sync` when a remote is + signed in (a `[cloud-folder]` remote with a token or client credentials in the volume's config), + else it reports "no source" and fails the status, and never pretends. The `mount` mode + (`rclone mount`) comes in a later version. Override without a rebuild with `CLOUD_FOLDER_MODE` + in `containerEnv`, or for one call with `task cloud-folder:status CLOUD_FOLDER_MODE=sync`. The + layout is fixed at build: with `mode` `sync` the mount point is a folder; with `auto` it stays the + link, and a sync fills the folder the link leads to (`/mnt/cloud-folder-host`, created at start). - `mountPoint` (default `/mnt/cloud-folder`): where the folder appears, an absolute path, and the - only place you name it. The install makes it a symbolic link to the host mount - `/mnt/cloud-folder-host` (no link when `mountPoint` is that path). It refuses, failing the - build with the reason, when `mountPoint` already exists in the image as a folder with content - or as a file; an empty folder or an old link is replaced. `CLOUD_FOLDER_MOUNT_POINT` in - `containerEnv` makes the status look at another path without a rebuild (no link is made then). - In the `sync` and `mount` modes of a later version the folder is filled at `mountPoint` - directly and no link is made. + only place you name it. In `auto` and `host` the install makes it a symbolic link to the host + mount and refuses, failing the build with the reason, when it already exists in the image as a + folder with content or as a file (an empty folder or an old link is replaced). In `sync` it is a + folder (a file there is refused). `CLOUD_FOLDER_MOUNT_POINT` in `containerEnv` makes the + scripts look at another path without a rebuild (no link or folder is made then). +- `remoteType` (default `sharepoint`): `sharepoint` syncs a document library of the site + `siteUrl`; it is the only kind the client-credentials sign-in of a cluster can use (ADR 0031). + `onedrive` syncs the signed-in person's own OneDrive. Both use rclone's `onedrive` backend. +- `driveType` (default `documentLibrary`): rclone's `drive_type`, written at login; the drive + lookup replaces it with what the drive reports. Change it only for `onedrive` (`personal` or + `business`) when the lookup is skipped (`CLOUD_FOLDER_DRIVE_ID`). +- `siteUrl` (default empty): the SharePoint site, for example + `https://.sharepoint.com/sites/`. Needed for `sharepoint` unless + `CLOUD_FOLDER_DRIVE_ID` names the drive. +- `remotePath` (default empty: the whole drive): the folder inside the library to sync, for + example `Shared/Recordings`. It must exist on the remote. Changing it gives the next sync a + `--resync`. +- `syncInterval` (default `15m`): how often the timer syncs: seconds, or a number with `s`, `m` + or `h`. `0`: no timer, sync on demand only. +- `rcloneVersion` (default `1.75.2`, the stable release on 2026-10-27, pinned with its sha256 per + architecture): another version, with or without a leading `v`, is verified against its + release's `SHA256SUMS`; `latest` resolves through `downloads.rclone.org/version.txt` (1.75.2 on + 2026-10-27) and the build log says what it resolved to. - `installTaskfile` (default `true`): install the Taskfile into the go-task registry. -The options of the decided design that belong to later versions (`remoteType`, `driveType`, -`siteUrl`, `remotePath`, `syncInterval`, `rcloneVersion`) are left out of this version rather than -accepted and ignored; the `sync` and `mount` modes come with them. +Every option but `rcloneVersion` and `installTaskfile` has a `CLOUD_FOLDER_*` variable that +overrides it without a rebuild (`CLOUD_FOLDER_MODE`, `CLOUD_FOLDER_MOUNT_POINT`, +`CLOUD_FOLDER_REMOTE_TYPE`, `CLOUD_FOLDER_DRIVE_TYPE`, `CLOUD_FOLDER_SITE_URL`, +`CLOUD_FOLDER_REMOTE_PATH`, `CLOUD_FOLDER_SYNC_INTERVAL`), from the task's command line, a `.env` +file or the environment, in that order. Login also reads `CLOUD_FOLDER_DRIVE_ID` (skip the drive +lookup) and the client-credentials variables below. ```json "features": { "ghcr.io/nabil-becker/devcontainer-features/go-task:1": {}, "ghcr.io/nabil-becker/devcontainer-features/cloud-folder:1": { - "mountPoint": "/mnt/cloud-folder" + "mountPoint": "/mnt/cloud-folder", + "siteUrl": "https://.sharepoint.com/sites/", + "remotePath": "" } }, "containerEnv": { @@ -106,76 +229,191 @@ accepted and ignored; the `sync` and `mount` modes come with them. } ``` +With `mode` left at `auto`, this one block serves both: a workstation with a synced host folder +gets `host`; a cluster (or any host without one) gets `sync` once someone has run +`task cloud-folder:login`. + ## State -None. This version keeps no state of its own: the folder's content lives on the host (and in -the cloud behind the sync client), and nothing else is stored. The config volume of the design -(`rclone.conf`, the cache, the sync state) comes with rclone in a later version. +The Feature's volume `cloud-folder-${devcontainerId}` at `/var/lib/devcontainer-features/cloud-folder` +(NFR-1), handed to the remote user by the post-start hook (`dcf-persist`) and made private (mode +700), holds: + +- `rclone.conf` (mode 600): the `[cloud-folder]` remote with its token or client secret. + `containerEnv` sets `RCLONE_CONFIG` to it, so a plain `rclone` in the container uses it too. + rclone rewrites it when it refreshes the token, so it must stay writable. +- `bisync/`: bisync's listings and lock files; `cache/`: rclone's cache. +- `run/loop.pid` (the timer), `run/sync.pid` (a running sync), `run/sync.started`. +- `last-sync` (the record of the last sync), `last-ok` (its time stamp when it succeeded), + `bisync-pair` (which remote and folder the listings are for). +- `sync.log` (the last sync's rclone log at INFO level: file names, no token; the one before is + `sync.log.1`) and `timer.log` (the timer's own lines, rotated at 1 MiB). + +The token is never put on a command line, in the workspace, in an image layer or in a log: login +reads it from stdin and writes it with shell builtins; rclone reads it from the config file; the +status reads only whether the key exists. A volume survives a rebuild and is deleted by a +container-engine storage reset; then `task cloud-folder:login` again. + +The synced folder itself (sync mode) is in the container's filesystem, not in the volume: a rebuild +empties it, and the next sync (a resync, since the marker is gone) fetches it again from the +remote. To keep the local copy across rebuilds, mount a named volume at `mountPoint` in +`devcontainer.json` and set `mode` `sync` explicitly (with `auto`, a mount there would be taken +for a host folder). In host mode the content lives on the host. ## Check it works ```sh -task cloud-folder:status # cloud-folder: ok: the folder is mounted and writable -task cloud-folder:status -- --json # one JSON object: mode, mount_point, real_path, linked, mounted, empty, writable, verdict, ok, hint +task cloud-folder:status # cloud-folder: ok: the folder is synced and writable (sync) + # cloud-folder: ok: the folder is mounted and writable (host) +task cloud-folder:status -- --json # one JSON object for agents +task cloud-folder:sync # cloud-folder: sync ok in 3s ``` -`status` prints the mode in use and why, whether the mount point is mounted, present but not -mounted, or missing, whether it is empty and writable, and what to do when it is not. It exits: - -- `0` when the folder is mounted and writable; an empty folder passes with a warning (right for - a new folder, a sign of a wrong path or an unfinished sync otherwise); -- `1` when nothing is mounted at the host mount (no source, or mode `host` with no mount), when the mount point - is a bare folder of the container, not a folder, not writable by this user, or the mode is not - one this version knows; so a repository's start-up check (`task cloud-folder:status` in its - own check) can stop before agents write into the wrong place; +`status` prints the mode in use and why, and then: + +- host mode: whether the mount point is mounted, present but not mounted, or missing, whether it + is empty and writable, and what to do when it is not; +- sync mode: the remote and whether it is configured (and how: token or client credentials, never + the token itself) and reachable (one `rclone lsjson --stat`, at most 30 s); the local folder, + whether it is empty and writable, and the number of files changed since the last sync; the last + sync's time and result; whether a sync and the timer are running. + +It exits: + +- `0` when the folder is usable: mounted and writable (host), or synced and writable (sync); an + empty folder passes with a warning; +- `1` when it is not: nothing mounted at the host mount (no source, or mode `host` with no mount), + a bare folder of the container, not a folder, not writable by this user; in sync mode no remote + signed in (`not-configured`), never synced in this folder (`never-synced`, or `syncing` while the + first sync runs), the last sync failed (`sync-failed`, with the first error line and what to do + about bisync's delete guard or an emptied folder), or bisync set its listings aside + (`needs-resync`); or the mode is not one this version knows. So a repository's start-up check + (`task cloud-folder:status` in its own check) can stop before agents write into the wrong + place. An unreachable remote is reported but does not fail the status on its own: the local + copy is still there; - `2` on an unknown argument. -It writes nothing into the folder: writability is the access check (`test -w`), so a status -never leaves a file for the sync client to upload. +The `--json` object has `mode`, `mount_point`, `real_path`, `linked`, `mounted`, `empty`, +`writable`, `verdict`, `ok`, `hint`, and in sync mode also `remote`, `remote_type`, `configured`, +`auth`, `remote_reachable`, `last_sync`, `last_result`, `last_rc`, `sync_running`, `sync_pid`, +`timer_running`, `timer_pid`, `interval` and `pending`. + +A status writes nothing into the folder: writability is the access check (`test -w`). The sync +writes the one hidden marker. ## Pairs with -`devcontainer-cli` (the launcher that adds the bind mount from `DCF_HOST_CLOUD_FOLDER`; its -bootstrap's `devcontainer.json` template maps the variable with `containerEnv`) and `go-task` -(for the tasks). It needs nothing else; `installsAfter` orders it behind `go-task` and -`common-utils`. +`devcontainer-cli` (the launcher that adds the host-mode bind mount from `DCF_HOST_CLOUD_FOLDER`; +its bootstrap's `devcontainer.json` template maps the variable with `containerEnv`) and `go-task` +(for the tasks). A repository's `shutdown:check` calls `cloud-folder-setup shutdown-check` (see +above). `installsAfter` orders it behind `go-task`, `common-utils` and the two community rclone +Features (`ghcr.io/devcontainers-extra/features/rclone`, +`ghcr.io/devcontainer-community/devcontainer-features/rclone.org`), so when a consumer uses one of +those, its rclone is installed first and kept. ## Tasks -`task cloud-folder:version`, `status` and `setup` (all in `cloud-folder`). `CLOUD_FOLDER_MODE` -and `CLOUD_FOLDER_MOUNT_POINT` on the command line, in a `.env` file or in the environment (in -that order) override the options for one call; `status` takes `-- --json`. +All in `cloud-folder`: + +- `version`: the Feature version, the mode option and the mode in use, the mount point, the state + folder and rclone's version. +- `status`: the report above; `-- --json` for agents. +- `setup`: re-run the post-start hook (the volume handed to this user, the report, the timer + started in sync mode); never fails. +- `login` (interactive): sign in, see "Sync mode". Without a prompt when the client-credentials + variables are set. +- `sync`: one bisync now; `-- --resync` forces a resync, `-- --force` lets a sync through bisync's + delete guard when the deletion is meant. +- `start`, `stop`: the timer. +- `shutdown-check`: for a repository's `shutdown:check`; `WAIT=`. + +`CLOUD_FOLDER_*` on the command line, in a `.env` file or in the environment (in that order) +override the options for one call. ## Limits and troubleshooting - **Host mode needs a host with a sync client.** It ties the container to one machine's synced - folder and path, and there is no host on a cluster; the `sync` and `mount` modes (rclone) of a - later version are the answer there. + folder and path; use sync mode where there is no such host. - **"no source" or "not-mounted"** with `DCF_HOST_CLOUD_FOLDER` set: the host folder did not exist when `up` ran (the launcher said so and skipped it), `HOST_MOUNTS=false`, or the container was started without the launcher (opened straight in VS Code). When the host mount - is mounted but `mountPoint` does not lead there, the report says so. Fix the cause and `task devcontainer:rebuild`: - mounts are fixed when the container is created. -- **"unwritable"**: the host folder's permissions, or the engine maps the mount read-only. -- **Files on demand.** A sync client that keeps files online-only shows them in the folder but - may fetch them only when opened, which is slow through a bind mount; mark the folder "always - keep on this device" on the host. -- Not tried yet on a real Windows host with a sync client; the tests fake the mount point. + is mounted but `mountPoint` does not lead there, the report says so. Fix the cause and + `task devcontainer:rebuild`: mounts are fixed when the container is created. +- **"unwritable"**: in host mode the host folder's permissions, or the engine maps the mount + read-only; in sync mode `task cloud-folder:setup` hands the folder to the remote user. +- **Files on demand** (host mode): a sync client that keeps files online-only shows them in the + folder but may fetch them only when opened; mark the folder "always keep on this device". +- **No device-code sign-in.** rclone documents only the browser flow and, headless, `rclone + authorize` on another machine (proposal finding 1); login follows that. +- **"Couldn't decode" or "not the token"** when pasting: the two rclone versions differ. Use the + version login prints on both sides. Login accepts the one-line JSON token that + `rclone authorize "onedrive"` prints (one argument); the base64 blob printed when `authorize` + gets a second argument is refused. +- **Sync is a copy, not a live view.** Changes on either side appear at the next sync (at most + `syncInterval` later). Two people changing the same file between syncs get both versions back + as `.conflict1` and `.conflict2`; merge them by hand and delete the extra. +- **What counts as pending** for the status and the shutdown check: files in the folder whose + modification time is after the start of the last successful sync. A deletion, a rename, or a + file copied in with its old time kept (`cp -p`) is not seen, so the check does not refuse a stop + for those; they are still synced by the next sync, which bisync's listings catch. A remote file + changed after the last sync started may be counted too (a false refusal, on the safe side). +- **The delete guard and an emptied folder.** A sync that would delete more than half the files of + one side, or sync from an empty side, stops and changes nothing; the status says `sync-failed` + with the reason. Restore the files, or when the deletion is meant, `task cloud-folder:sync -- + --force`; an emptied folder by mistake comes back with `-- --resync`. +- **A hard kill** (a stop with `FORCE=1`, a crash) during a sync leaves bisync's lock (it expires + after 5 minutes) and possibly a half-done run; `--recover` picks it up on the next sync. When + bisync still sets its listings aside (`.lst-err`), the status says `needs-resync`: read + `sync.log`, then `task cloud-folder:sync -- --resync`. +- **The timer and a stop.** The shutdown check refuses while a sync runs, but does not pause the + timer: a round can start between a passing check and the stop; it is then interrupted as above. +- **The timer's processes** are children of the container's first process once their starter + exits; a container without an init that reaps them keeps the ended ones as zombies until it + stops (harmless; `init: true` in `devcontainer.json` avoids it). +- **Unverified (no SharePoint was available when this was built; the tests use a local remote):** + - the drive lookup against a real site (rclone's non-interactive config with `config_type=url`; + exercised here only up to the Graph call, which a fake token got refused); + - **the client-credentials sign-in** for a cluster: `CLOUD_FOLDER_CLIENT_ID`, + `CLOUD_FOLDER_TENANT` and `CLOUD_FOLDER_CLIENT_SECRET_FILE` (a file the platform mounts) or + `CLOUD_FOLDER_CLIENT_SECRET`, written into the config without being printed; whether + `Sites.Selected` works end to end and which `drive_id` it needs (proposal "Still open" 1, 2). + Login runs it on request only; the post-start hook does not sign in on its own; + - bisync with SharePoint's QuickXorHash and 1-second modification times (proposal "Still open" + 3): whether the default comparison gives false conflicts; + - rclone with a read-only config (a secret mounted read-only): not supported; the config must + be in the writable volume (proposal "Still open" 5); + - the graceful stop of a real bisync by SIGINT from the timer (bisync's documented "Graceful + Shutdown"; the tests use a fake). ## Why it exists -The two Features for rclone published on containers.dev (checked 2026-10-27) install the -binary only: none configures a remote, signs in, mounts or syncs, and none covers OneDrive, -SharePoint or a host folder. This Feature gives the folder one fixed path and a status a -start-up check can trust, whatever fills it; rclone comes in a later version, installed only when -absent and ordered after those Features. +The two Features for rclone published on containers.dev (checked 2026-10-27: +`ghcr.io/devcontainers-extra/features/rclone` and +`ghcr.io/devcontainer-community/devcontainer-features/rclone.org`) install the binary only: none +configures a remote, signs in, mounts or syncs, and none covers OneDrive, SharePoint or a host +folder. This Feature gives the folder one fixed path, fills it from a host folder or with rclone +bisync, keeps the sign-in in its own volume, and has a status and a shutdown check a repository +can trust. It installs rclone only when absent and orders itself after those Features. ## Version and upgrade -`1.0.0`: the first release, host mode only (`mode` `auto` or `host`, `mountPoint` linked to the -host mount `/mnt/cloud-folder-host`, the `status`, `setup` and `version` tasks). The host side -lives in `devcontainer-cli` 1.12.0 or later (`DCF_HOST_CLOUD_FOLDER`). Pin the Feature with `:1`. A later minor -version adds the rclone modes and their options; `host` and the mount point stay as they are. +- `1.1.0`: sync mode (rclone pinned and verified, `login`, `sync`, the timer, `start`, `stop`, + `shutdown-check`), the options `remoteType`, `driveType`, `siteUrl`, `remotePath`, + `syncInterval` and `rcloneVersion`, the Feature volume, and `auto` choosing sync when a remote + is signed in. Host mode and the mount point are unchanged. An image built with `1.0.0` gains + rclone (unless `mode` is `host`) and the volume at the next rebuild. +- `1.0.0`: host mode only. + +The host side lives in `devcontainer-cli` 1.12.0 or later (`DCF_HOST_CLOUD_FOLDER`). Pin the +Feature with `:1`. A later minor version adds the `mount` mode. + +To pin another rclone version without a network lookup at build, add its two digests to +`rclone-checksums.txt` (check them against both the publisher's and the GitHub release's +`SHA256SUMS`): + +```sh +v=1.75.2; curl -fsSL https://downloads.rclone.org/v$v/SHA256SUMS | grep -E "rclone-v$v-linux-(amd64|arm64)\.zip$" +``` --- diff --git a/src/cloud-folder/cloud-folder-setup.sh b/src/cloud-folder/cloud-folder-setup.sh index 2ef500b..546c956 100644 --- a/src/cloud-folder/cloud-folder-setup.sh +++ b/src/cloud-folder/cloud-folder-setup.sh @@ -1,30 +1,54 @@ #!/usr/bin/env bash # cloud-folder-setup: the cloud-folder Feature's postStartCommand (a host folder is mounted, or -# not, per start), also behind `task cloud-folder:setup`, `task cloud-folder:status` and -# `task cloud-folder:version`. +# not, per start; the sync timer is started per start), also behind every `task cloud-folder:*`. # -# cloud-folder-setup report on the folder (never fatal: exit 0) -# cloud-folder-setup status [--json] the same report; exit 0 when the folder is mounted and -# writable (empty or not), 1 when it is missing, not -# mounted, not writable, or the mode is not one this -# version knows. --json prints one JSON object instead. -# cloud-folder-setup version the Feature version and the mode and mount point in use +# cloud-folder-setup post-start: make the state volume usable, report on the +# folder, start the sync timer in sync mode (never fatal) +# cloud-folder-setup status [--json] the report; exit 0 when the folder is usable, 1 when not +# (host: missing, not mounted, not writable; sync: no +# remote, never synced, last sync failed, not writable; or +# a mode this version does not know). --json: one object. +# cloud-folder-setup version the Feature version, the mode, the mount point, rclone +# cloud-folder-setup login sign in (interactive: a token pasted from `rclone +# authorize "onedrive"`; or client credentials from +# CLOUD_FOLDER_CLIENT_* without a prompt) and write the +# remote into the Feature's rclone.conf; then start the timer +# cloud-folder-setup sync [--resync] [--force] +# one `rclone bisync` between the remote and the folder +# (the first run of a folder is a --resync on its own) +# cloud-folder-setup start | stop the sync timer (one supervised loop, PID file in the +# volume; start is idempotent, stop waits for a running sync) +# cloud-folder-setup run the timer loop itself (start runs it detached) +# cloud-folder-setup shutdown-check [SECONDS] +# for a repository's shutdown:check: exit 1 while a sync +# runs (after waiting up to SECONDS, default 60) or while +# local files changed since the last successful sync # cloud-folder-setup link MOUNT_POINT HOST_MOUNT # install.sh, as root: make MOUNT_POINT a symbolic link to # HOST_MOUNT (nothing when they are the same path) +# cloud-folder-setup folder MOUNT_POINT +# install.sh, as root, mode sync: make MOUNT_POINT a folder # -# This version knows host mode only: the devcontainer-cli launcher bind-mounts the host folder -# named by DCF_HOST_CLOUD_FOLDER at one fixed path, the host mount (/mnt/cloud-folder-host), and -# install.sh links the option mountPoint to it when the two differ, so a consumer names the path -# once. "Mounted" means the path the mount point leads to (through the link) is the target of a -# mount in /proc/self/mountinfo. A folder that exists but is not mounted is a bare folder of the -# container: writing there would put files where no one else sees them, so the status fails +# Host mode: the devcontainer-cli launcher bind-mounts the host folder named by +# DCF_HOST_CLOUD_FOLDER at one fixed path, the host mount (/mnt/cloud-folder-host), and +# install.sh links the option mountPoint to it when the two differ. "Mounted" means the path the +# mount point leads to (through the link) is the target of a mount in /proc/self/mountinfo. A +# folder that exists but is not mounted is a bare folder of the container: the status fails # rather than call it a cloud folder. # -# Nothing is written into the folder: writability is the access check (test -w), so a status -# never leaves a file behind for a sync client to upload. +# Sync mode: no link and no mount; `rclone bisync` fills the folder the mount point leads to +# (path 1 the remote cloud-folder:, path 2 that folder). The rclone config (and so the +# token), the bisync listings, the PID files, the last-sync record and the logs live in the +# Feature's volume (STATE_DIR), never in the workspace or the image; the token is never put on a +# command line and never printed. rclone reads it from the config file (RCLONE_CONFIG). +# +# Nothing is written into the folder by a status: writability is the access check (test -w). +# The sync writes one hidden marker there (.cloud-folder-bisync, excluded from the sync) that +# tells a folder already synced from a new one (after a rebuild), which gets a --resync. # Test hooks (not options): DCF_CLOUD_FOLDER_MOUNTINFO names the mount table to read (default -# /proc/self/mountinfo); DCF_CLOUD_FOLDER_HOST_MOUNT replaces the fixed host mount path. +# /proc/self/mountinfo); DCF_CLOUD_FOLDER_HOST_MOUNT replaces the fixed host mount path; +# DCF_CLOUD_FOLDER_STATE_DIR replaces the state volume's path; DCF_CLOUD_FOLDER_GRACE the +# seconds a sync gets to stop gracefully (default 100). set -u # Defaults baked by install.sh from the Feature options; CLOUD_FOLDER_* in the environment @@ -37,12 +61,35 @@ feature_version="${FEATURE_VERSION:-unknown}" host_mount="${DCF_CLOUD_FOLDER_HOST_MOUNT:-${HOST_MOUNT:-/mnt/cloud-folder-host}}" mountinfo="${DCF_CLOUD_FOLDER_MOUNTINFO:-/proc/self/mountinfo}" host_source="${DCF_HOST_CLOUD_FOLDER:-}" +remote_type="${CLOUD_FOLDER_REMOTE_TYPE:-${REMOTETYPE:-sharepoint}}" +drive_type="${CLOUD_FOLDER_DRIVE_TYPE:-${DRIVETYPE:-documentLibrary}}" +site_url="${CLOUD_FOLDER_SITE_URL:-${SITEURL:-}}" +remote_path="${CLOUD_FOLDER_REMOTE_PATH:-${REMOTEPATH:-}}" +sync_interval="${CLOUD_FOLDER_SYNC_INTERVAL:-${SYNCINTERVAL:-15m}}" +rclone_version_option="${RCLONEVERSION:-}" +state_dir="${DCF_CLOUD_FOLDER_STATE_DIR:-${STATE_DIR:-/var/lib/devcontainer-features/cloud-folder}}" +grace="${DCF_CLOUD_FOLDER_GRACE:-100}" + +conf="$state_dir/rclone.conf" +run_dir="$state_dir/run" +remote_name=cloud-folder +remote="$remote_name:$remote_path" +marker_name=.cloud-folder-bisync +self="$(readlink -f -- "${BASH_SOURCE[0]}" 2>/dev/null || printf '%s' "$0")" # A trailing slash would never match the mount table. while [ "${#mount_point}" -gt 1 ] && [ "${mount_point%/}" != "$mount_point" ]; do mount_point="${mount_point%/}"; done say() { echo "cloud-folder: $*"; } +# rcl : rclone with the Feature's config and cache in the volume. Only paths and flags go +# on its command line, never a secret. +rcl() { RCLONE_CONFIG="$conf" RCLONE_CACHE_DIR="$state_dir/cache" rclone "$@"; } + +as_root() { # run a command as root: directly, or through passwordless sudo + if [ "$(id -u)" -eq 0 ]; then "$@"; elif command -v sudo >/dev/null 2>&1 && sudo -n true 2>/dev/null; then sudo -n "$@"; else return 1; fi +} + # is_mounted : the path is the target of a mount. The mount table escapes space, tab, # newline and backslash as octal (\040, \011, \012, \134), so the path is encoded the same way. is_mounted() { @@ -83,34 +130,169 @@ link_mount_point() { say "mountPoint $mp links to the host mount $hm" } -# assess: sets MODE_USED (host | none), WHY, REAL (where the mount point leads), LINKED, PRESENT, -# MOUNTED, IS_DIR, EMPTY (yes | no | unknown), WRITABLE, VERDICT (ok | empty | not-mounted | -# no-source | not-a-folder | unwritable | unknown-mode) and HINT. -assess() { - PRESENT=no; MOUNTED=no; IS_DIR=no; EMPTY=unknown; WRITABLE=no; HINT=''; LINKED=no +# make_folder : the build-time folder of mode sync (install.sh runs it as root). A +# link (left by an earlier layout) gives way; a folder stays as it is; any other file is refused. +make_folder() { + local mp="$1" + case "$mp" in /?*) ;; *) echo "cloud-folder: mountPoint '$mp' must be an absolute path." >&2; return 1 ;; esac + while [ "${#mp}" -gt 1 ] && [ "${mp%/}" != "$mp" ]; do mp="${mp%/}"; done + if [ -L "$mp" ]; then rm -f -- "$mp" || return 1; fi + if [ -e "$mp" ] && [ ! -d "$mp" ]; then + echo "cloud-folder: mountPoint $mp exists in the image and is not a folder; the sync mode needs a folder there." >&2 + return 1 + fi + mkdir -p -- "$mp" || return 1 + # Hand it to the remote user now when the build names one; the post-start report does it again. + if [ -n "${_REMOTE_USER:-}" ] && [ "${_REMOTE_USER}" != root ] && id -u "$_REMOTE_USER" >/dev/null 2>&1; then + chown "$_REMOTE_USER:$(id -gn "$_REMOTE_USER")" "$mp" || return 1 + fi + say "mountPoint $mp is a folder, filled by the sync (it is handed to the remote user at start)" +} + +# ------------------------------------------------------------- processes ---- +# alive : a running process (a zombie, gone but not yet reaped, does not count). +alive() { + local st + case "${1:-}" in '' | *[!0-9]*) return 1 ;; esac + kill -0 "$1" 2>/dev/null || return 1 + st="$(cat "/proc/$1/stat" 2>/dev/null)" || return 0 + st="${st##*) }" + [ "${st%% *}" != Z ] +} +# ours : alive and a cloud-folder-setup process (a PID file may outlive its process, and the +# number be reused). +ours() { + alive "${1:-}" || return 1 + tr '\0' ' ' < "/proc/$1/cmdline" 2>/dev/null | grep -q cloud-folder-setup +} +pid_of() { local p=''; [ -f "$1" ] && read -r p < "$1" 2>/dev/null; case "$p" in '' | *[!0-9]*) echo '' ;; *) echo "$p" ;; esac; } +# take_lock : write this shell's PID atomically; a file whose process is gone is stale +# and replaced. Exit 1 when another live process holds it. +take_lock() { + if (set -o noclobber; echo "$$" > "$1") 2>/dev/null; then return 0; fi + ours "$(pid_of "$1")" && return 1 + rm -f -- "$1" + (set -o noclobber; echo "$$" > "$1") 2>/dev/null +} +# shellcheck disable=SC2329 # called from the EXIT traps +drop_lock() { [ "$(pid_of "$1")" = "$$" ] && rm -f -- "$1"; return 0; } + +# interval_seconds: syncInterval in seconds (15m, 900s, 900, 1h; 0 = on demand only). +interval_seconds() { + local n u + [[ "$sync_interval" =~ ^([0-9]+)([smh]?)$ ]] || return 1 + n="${BASH_REMATCH[1]}"; u="${BASH_REMATCH[2]}" + n=$((10#$n)) + case "$u" in m) n=$((n * 60)) ;; h) n=$((n * 3600)) ;; esac + echo "$n" +} + +# ----------------------------------------------------------------- state ---- +# prepare_state: the volume usable by this user (dcf-persist, as every Feature of the collection +# does), the folders inside it private, the config mode 600. Quiet unless asked (report). +prepare_state() { + local verbose="${1:-quiet}" + if command -v dcf-persist >/dev/null 2>&1; then + if [ "$verbose" = verbose ]; then dcf-persist --label 'cloud-folder state' "$state_dir"; else dcf-persist "$state_dir" >/dev/null 2>&1; fi + elif [ ! -d "$state_dir" ]; then + mkdir -p -- "$state_dir" 2>/dev/null || { as_root mkdir -p -- "$state_dir" && as_root chown "$(id -u):$(id -g)" "$state_dir"; } + fi + [ -w "$state_dir" ] || return 1 + chmod 0700 "$state_dir" 2>/dev/null + (umask 077; mkdir -p -- "$run_dir" "$state_dir/bisync" "$state_dir/cache") || return 1 + [ -f "$conf" ] && chmod 0600 "$conf" 2>/dev/null + return 0 +} + +# conf_state: CONF (missing | unreadable | no-remote | no-token | ok), CONF_TYPE and CONF_AUTH +# (token | client-credentials | none), read from the config without printing any value. +conf_state() { + local out found tok cc + CONF=missing; CONF_TYPE=''; CONF_AUTH=none + [ -e "$conf" ] || return 0 + if [ ! -r "$conf" ]; then CONF=unreadable; return 0; fi + out="$(awk -v sec="[$remote_name]" ' + /^[[:space:]]*\[/ { s = $0; gsub(/[[:space:]]/, "", s); in_s = (s == sec); if (in_s) found = 1; next } + in_s && /^[[:space:]]*type[[:space:]]*=/ { t = $0; sub(/^[^=]*=[[:space:]]*/, "", t); sub(/[[:space:]]*$/, "", t); type = t } + in_s && /^[[:space:]]*token[[:space:]]*=[[:space:]]*[^[:space:]]/ { tok = 1 } + in_s && /^[[:space:]]*client_credentials[[:space:]]*=[[:space:]]*true/ { cc = 1 } + END { printf "%d %s %d %d\n", found, (type == "" ? "-" : type), tok, cc }' "$conf" 2>/dev/null)" + read -r found CONF_TYPE tok cc <<<"$out" + [ "$CONF_TYPE" = - ] && CONF_TYPE='' + if [ "${found:-0}" != 1 ]; then CONF=no-remote; return 0; fi + if [ "${cc:-0}" = 1 ]; then CONF_AUTH=client-credentials; elif [ "${tok:-0}" = 1 ]; then CONF_AUTH=token; fi + if [ "$CONF_TYPE" = onedrive ] && [ "$CONF_AUTH" = none ]; then CONF=no-token; return 0; fi + CONF=ok +} + +# read_record: the last sync, from the record the sync writes (key=value lines, never sourced). +read_record() { + local k v + LAST_TIME=''; LAST_RESULT=''; LAST_RC=''; LAST_RESYNC=''; LAST_SECONDS=''; LAST_NOTE='' + [ -r "$state_dir/last-sync" ] || return 0 + while IFS='=' read -r k v; do + case "$k" in + time) LAST_TIME="$v" ;; result) LAST_RESULT="$v" ;; rc) LAST_RC="$v" ;; + resync) LAST_RESYNC="$v" ;; seconds) LAST_SECONDS="$v" ;; note) LAST_NOTE="$v" ;; + esac + done < "$state_dir/last-sync" +} + +# pending_count : files changed in the folder since the last successful sync started +# (all files when there was none). Deletions and files arriving with an old modification time +# are not seen; see the NOTES. +pending_count() { + local dir="$1" + [ -d "$dir" ] && [ -r "$dir" ] || { echo 0; return; } + if [ -f "$state_dir/last-ok" ]; then + find -H "$dir" -type f ! -name "$marker_name" -newer "$state_dir/last-ok" -print 2>/dev/null | head -n 10000 | wc -l | tr -d ' ' + else + find -H "$dir" -type f ! -name "$marker_name" -print 2>/dev/null | head -n 10000 | wc -l | tr -d ' ' + fi +} + +# ---------------------------------------------------------------- assess ---- +# resolve: sets the host facts (REAL, LINKED, PRESENT, MOUNTED, IS_DIR, EMPTY, WRITABLE) and the +# mode: MODE_USED (host | sync | none) and WHY, or VERDICT=unknown-mode. +resolve() { + PRESENT=no; MOUNTED=no; IS_DIR=no; EMPTY=unknown; WRITABLE=no; HINT=''; LINKED=no; VERDICT='' REAL="$(real_path "$mount_point")" [ -L "$mount_point" ] && LINKED=yes [ -e "$mount_point" ] && PRESENT=yes [ -d "$mount_point" ] && IS_DIR=yes is_mounted "$REAL" && MOUNTED=yes if [ "$IS_DIR" = yes ] && [ -r "$mount_point" ] && [ -x "$mount_point" ]; then - if [ -n "$(find -H "$mount_point" -mindepth 1 -maxdepth 1 -print -quit 2>/dev/null)" ]; then EMPTY=no; else EMPTY=yes; fi + if [ -n "$(find -H "$mount_point" -mindepth 1 -maxdepth 1 ! -name "$marker_name" -print -quit 2>/dev/null)" ]; then EMPTY=no; else EMPTY=yes; fi fi [ "$IS_DIR" = yes ] && [ -w "$mount_point" ] && [ -x "$mount_point" ] && WRITABLE=yes + conf_state case "$mode_option" in host) MODE_USED=host; WHY='set by the option or CLOUD_FOLDER_MODE' ;; + sync) + MODE_USED=sync; WHY='set by the option or CLOUD_FOLDER_MODE' ;; auto) if [ "$MOUNTED" = yes ]; then MODE_USED=host; WHY="auto: something is mounted at $mount_point" - else MODE_USED=none; WHY="auto: nothing is mounted at $mount_point, and this version has no other source"; fi ;; - sync | mount) - MODE_USED=none; WHY="mode $mode_option is not in this version of the Feature ($feature_version), which knows auto and host; the rclone modes come later" - VERDICT=unknown-mode; return 0 ;; + elif [ "$CONF" = ok ]; then MODE_USED=sync; WHY="auto: nothing is mounted at $mount_point, and a remote is configured" + else MODE_USED=none; WHY="auto: nothing is mounted at $mount_point, and no remote is configured"; fi ;; + mount) + MODE_USED=none; WHY="mode mount is not in this version of the Feature ($feature_version), which knows auto, host and sync; it comes later" + VERDICT=unknown-mode ;; *) - MODE_USED=none; WHY="mode '$mode_option' is unknown (auto or host)" - VERDICT=unknown-mode; return 0 ;; + MODE_USED=none; WHY="mode '$mode_option' is unknown (auto, host or sync)" + VERDICT=unknown-mode ;; esac +} + +# assess [remote]: resolve, then the verdict (ok | empty | not-mounted | no-source | not-a-folder +# | unwritable | unknown-mode, and in sync mode not-configured | no-rclone | never-synced | +# syncing | needs-resync | sync-failed) and HINT. With "remote", sync mode asks the remote +# whether it answers (REACHABLE yes | no | unknown). +assess() { + resolve + [ "$VERDICT" = unknown-mode ] && return 0 + if [ "$MODE_USED" = sync ]; then assess_sync "${1:-}"; return 0; fi if [ "$MOUNTED" = no ]; then if [ "$MODE_USED" = none ]; then VERDICT=no-source; else VERDICT=not-mounted; fi @@ -119,6 +301,9 @@ assess() { else HINT="no host folder: set DCF_HOST_CLOUD_FOLDER= in .devcontainer/.env on the host and start with task devcontainer:up; the launcher then mounts it here (map DCF_HOST_CLOUD_FOLDER with containerEnv so this report can name it)." fi + if [ "$MODE_USED" = none ]; then + HINT="$HINT Or sync it with rclone: task cloud-folder:login, then the timer (or task cloud-folder:sync) fills it." + fi if [ "$REAL" != "$host_mount" ] && is_mounted "$host_mount"; then HINT="$HINT The host mount $host_mount is mounted, but $mount_point does not lead to it (CLOUD_FOLDER_MOUNT_POINT names another path than the option mountPoint, or the image predates the link): use $host_mount, or rebuild." fi @@ -141,6 +326,60 @@ assess() { VERDICT=ok } +assess_sync() { + local check_remote="${1:-}" p err + read_record + SYNC_PID=''; LOOP_PID=''; REACHABLE=unknown; REACH_ERR='' + p="$(pid_of "$run_dir/sync.pid")"; ours "$p" && SYNC_PID="$p" + p="$(pid_of "$run_dir/loop.pid")"; ours "$p" && LOOP_PID="$p" + RCLONE_OK=no; command -v rclone >/dev/null 2>&1 && RCLONE_OK=yes + MARKER=no; [ -f "$REAL/$marker_name" ] && MARKER=yes + PENDING="$(pending_count "$REAL")" + NEEDS_RESYNC=no + [ -n "$(find "$state_dir/bisync" -maxdepth 1 -name '*.lst-err' -print -quit 2>/dev/null)" ] && NEEDS_RESYNC=yes + if [ "$check_remote" = remote ] && [ "$CONF" = ok ] && [ "$RCLONE_OK" = yes ]; then + if err="$(timeout 30 env RCLONE_CONFIG="$conf" RCLONE_CACHE_DIR="$state_dir/cache" rclone lsjson --stat --contimeout 10s --timeout 15s --low-level-retries 1 --retries 1 "$remote" 2>&1 >/dev/null)"; then + REACHABLE=yes + else + REACHABLE=no; REACH_ERR="$(printf '%s\n' "$err" | grep -v '^[[:space:]]*$' | tail -n 1 | cut -c1-300)" + fi + fi + + case "$CONF" in + ok) ;; + unreadable) VERDICT=not-configured; HINT="$conf cannot be read by $(id -un 2>/dev/null || id -u): run task cloud-folder:setup (it hands the volume to this user)"; return 0 ;; + *) VERDICT=not-configured; HINT="no remote '$remote_name' with a sign-in in $conf: run task cloud-folder:login"; return 0 ;; + esac + if [ "$RCLONE_OK" = no ]; then VERDICT=no-rclone; HINT='rclone is not on PATH: rebuild with the Feature in mode auto or sync'; return 0; fi + if [ "$PRESENT" = yes ] && [ "$IS_DIR" = no ]; then VERDICT=not-a-folder; HINT="$mount_point is not a folder"; return 0; fi + if [ "$IS_DIR" = yes ] && [ "$WRITABLE" = no ]; then + VERDICT=unwritable; HINT="$(id -un 2>/dev/null || id -u) cannot write to $mount_point: run task cloud-folder:setup (it hands the folder to this user)"; return 0 + fi + if [ "$NEEDS_RESYNC" = yes ]; then + VERDICT=needs-resync; HINT="bisync stopped on a critical error and keeps its listings aside (.lst-err in $state_dir/bisync): read $state_dir/sync.log, then task cloud-folder:sync -- --resync"; return 0 + fi + if [ -z "$LAST_RESULT" ] || [ "$MARKER" = no ]; then + if [ -n "$SYNC_PID" ]; then VERDICT=syncing; HINT="the first sync of $REAL is running (pid $SYNC_PID): wait for it" + else + VERDICT=never-synced + if [ -n "$LAST_RESULT" ]; then HINT="$REAL has not been synced yet (a new folder, after a rebuild?): task cloud-folder:sync (or wait for the timer) fills it" + else HINT="no sync yet: task cloud-folder:sync (or wait for the timer)"; fi + fi + return 0 + fi + if [ "$LAST_RESULT" != ok ]; then + VERDICT=sync-failed + HINT="the last sync ($LAST_TIME) ended $LAST_RESULT (rclone exit $LAST_RC)${LAST_NOTE:+: $LAST_NOTE}; log: $state_dir/sync.log" + case "$LAST_NOTE" in + *'too many deletes'*) HINT="$HINT. bisync's delete guard stopped it (more than half the files of one side are gone): restore them, or, if the deletion is meant, task cloud-folder:sync -- --force" ;; + *'Empty current Path'*) HINT="$HINT. One side is empty and bisync will not empty the other: if the folder was emptied by mistake, task cloud-folder:sync -- --resync brings the files back" ;; + esac + return 0 + fi + if [ "$EMPTY" = yes ]; then VERDICT=empty; HINT="synced, but the folder is empty: right for an empty remote folder; otherwise check remotePath"; return 0; fi + VERDICT=ok +} + json_str() { # : a JSON string literal local s="$1" s="${s//\\/\\\\}"; s="${s//\"/\\\"}" @@ -148,23 +387,31 @@ json_str() { # : a JSON string literal printf '"%s"' "$s" } json_bool() { if [ "$1" = yes ]; then printf true; else printf false; fi; } +json_tri() { case "$1" in yes) printf true ;; no) printf false ;; *) printf null ;; esac; } + +verdict_ok() { [ "$VERDICT" = ok ] || [ "$VERDICT" = empty ]; } print_json() { - local empty - case "$EMPTY" in yes) empty=true ;; no) empty=false ;; *) empty=null ;; esac printf '{"mode_option":%s,"mode":%s,"why":%s,"mount_point":%s,"real_path":%s,"linked":%s,"host_source":%s,' \ "$(json_str "$mode_option")" "$(json_str "$MODE_USED")" "$(json_str "$WHY")" "$(json_str "$mount_point")" \ "$(json_str "$REAL")" "$(json_bool "$LINKED")" "$(json_str "$host_source")" - printf '"present":%s,"mounted":%s,"folder":%s,"empty":%s,"writable":%s,"verdict":%s,"ok":%s,"hint":%s}\n' \ - "$(json_bool "$PRESENT")" "$(json_bool "$MOUNTED")" "$(json_bool "$IS_DIR")" "$empty" "$(json_bool "$WRITABLE")" \ + printf '"present":%s,"mounted":%s,"folder":%s,"empty":%s,"writable":%s,' \ + "$(json_bool "$PRESENT")" "$(json_bool "$MOUNTED")" "$(json_bool "$IS_DIR")" "$(json_tri "$EMPTY")" "$(json_bool "$WRITABLE")" + if [ "$MODE_USED" = sync ]; then + printf '"remote":%s,"remote_type":%s,"configured":%s,"auth":%s,"remote_reachable":%s,"last_sync":%s,"last_result":%s,"last_rc":%s,"sync_running":%s,"sync_pid":%s,"timer_running":%s,"timer_pid":%s,"interval":%s,"pending":%s,' \ + "$(json_str "$remote")" "$(json_str "$remote_type")" "$(json_bool "$([ "$CONF" = ok ] && echo yes)")" "$(json_str "$CONF_AUTH")" \ + "$(json_tri "$REACHABLE")" "$(json_str "$LAST_TIME")" "$(json_str "$LAST_RESULT")" "${LAST_RC:-null}" \ + "$(json_bool "$([ -n "$SYNC_PID" ] && echo yes)")" "${SYNC_PID:-null}" "$(json_bool "$([ -n "$LOOP_PID" ] && echo yes)")" "${LOOP_PID:-null}" \ + "$(json_str "$sync_interval")" "${PENDING:-0}" + fi + printf '"verdict":%s,"ok":%s,"hint":%s}\n' \ "$(json_str "$VERDICT")" "$(if verdict_ok; then printf true; else printf false; fi)" "$(json_str "$HINT")" } -verdict_ok() { [ "$VERDICT" = ok ] || [ "$VERDICT" = empty ]; } - print_report() { local state say "mode $MODE_USED ($WHY)" + if [ "$MODE_USED" = sync ]; then print_sync_report; return; fi if [ "$MOUNTED" = yes ]; then state=mounted; elif [ "$PRESENT" = yes ]; then state='present, not mounted'; else state=missing; fi if [ "$LINKED" = yes ]; then say "mount point $mount_point -> $REAL: $state" else say "mount point $mount_point: $state"; fi @@ -173,19 +420,402 @@ print_report() { case "$EMPTY" in yes) state=empty ;; no) state='not empty' ;; *) state='cannot be listed' ;; esac say "contents: $state; writable: $WRITABLE" fi + print_verdict +} + +print_sync_report() { + local state auth reach timer + case "$CONF_AUTH" in token) auth='signed in with a token' ;; client-credentials) auth='client credentials' ;; *) auth='no sign-in' ;; esac + case "$CONF" in ok) state="configured, $auth" ;; *) state="not configured ($CONF)" ;; esac + case "$REACHABLE" in yes) reach=reachable ;; no) reach="NOT reachable${REACH_ERR:+: $REACH_ERR}" ;; *) reach='reachability not checked' ;; esac + say "remote $remote (remoteType $remote_type${site_url:+, site $site_url}): $state; $reach" + if [ "$PRESENT" = yes ]; then state=present; else state='missing (made at the first sync)'; fi + if [ "$LINKED" = yes ]; then say "local copy $mount_point -> $REAL: $state"; else say "local copy $mount_point: $state"; fi + if [ "$IS_DIR" = yes ]; then + case "$EMPTY" in yes) state=empty ;; no) state='not empty' ;; *) state='cannot be listed' ;; esac + say "contents: $state; writable: $WRITABLE; files changed since the last sync: ${PENDING:-0}" + fi + if [ -n "$LAST_RESULT" ]; then say "last sync: $LAST_TIME, $LAST_RESULT (rclone exit $LAST_RC${LAST_SECONDS:+, ${LAST_SECONDS}s}${LAST_RESYNC:+, resync $LAST_RESYNC})" + else say "last sync: never"; fi + if [ -n "$LOOP_PID" ]; then timer="running (pid $LOOP_PID, every $sync_interval)" + elif [ "$(interval_seconds 2>/dev/null)" = 0 ]; then timer='off (syncInterval 0: on demand only)' + else timer='not running'; fi + if [ -n "$SYNC_PID" ]; then say "sync: running now (pid $SYNC_PID); timer: $timer"; else say "sync: not running; timer: $timer"; fi + print_verdict +} + +print_verdict() { case "$VERDICT" in - ok) say "ok: the folder is mounted and writable" ;; - empty) say "ok, but empty: ${HINT#the folder is mounted and writable but empty: }" ;; + ok) if [ "$MODE_USED" = sync ]; then say "ok: the folder is synced and writable"; else say "ok: the folder is mounted and writable"; fi ;; + empty) say "ok, but empty: ${HINT#*: }" ;; *) say "not usable ($VERDICT): ${HINT:-$WHY}" ;; esac } +# ------------------------------------------------------------------ sync ---- +# ensure_local : the sync's local folder exists and this user can write it. +ensure_local() { + local d="$1" + if [ ! -d "$d" ]; then + mkdir -p -- "$d" 2>/dev/null || as_root mkdir -p -- "$d" || { say "cannot create $d"; return 1; } + fi + if [ ! -w "$d" ] || [ "$(stat -c %u "$d" 2>/dev/null)" != "$(id -u)" ]; then + as_root chown "$(id -u):$(id -g)" "$d" 2>/dev/null || true + fi + [ -w "$d" ] || { say "$(id -un 2>/dev/null || id -u) cannot write to $d"; return 1; } +} + +# do_sync [--resync] [--force] [--timer]: one bisync. Exit 0 ok; 1 failed (a later run may +# succeed); 7 critical (bisync wants a --resync); 2 usage; 3 cannot sync here (mode, sign-in, +# rclone, folder); 4 another sync is running. +do_sync() { + local resync=no force=no timer=no why_resync='' a child rc start secs result note pair stopping=no i + for a in "$@"; do + case "$a" in + --resync) resync=yes; why_resync='asked for' ;; + --force) force=yes ;; + --timer) timer=yes ;; + *) echo "cloud-folder: sync: unknown argument '$a' (--resync, --force)" >&2; return 2 ;; + esac + done + resolve + if [ "$MODE_USED" != sync ]; then say "nothing to sync: mode $MODE_USED ($WHY)"; return 3; fi + if [ "$CONF" != ok ]; then say "no remote to sync with ($CONF): run task cloud-folder:login"; return 3; fi + command -v rclone >/dev/null 2>&1 || { say 'rclone is not on PATH'; return 3; } + prepare_state || { say "the state folder $state_dir is not writable"; return 3; } + ensure_local "$REAL" || return 3 + if ! take_lock "$run_dir/sync.pid"; then + if [ "$timer" = yes ]; then say "timer: skipped, a sync is already running (pid $(pid_of "$run_dir/sync.pid"))" + else say "a sync is already running (pid $(pid_of "$run_dir/sync.pid")); try again when it ends (task cloud-folder:status)"; fi + return 4 + fi + # shellcheck disable=SC2064 # the lock path is fixed now + trap "drop_lock '$run_dir/sync.pid'" EXIT + + pair="$remote -> $REAL" + if [ "$resync" = no ]; then + if [ ! -f "$REAL/$marker_name" ]; then resync=yes; why_resync="first sync of $REAL" + elif [ "$(cat "$state_dir/bisync-pair" 2>/dev/null)" != "$pair" ]; then resync=yes; why_resync='the remote or the folder changed'; fi + fi + + local -a args=(bisync "$remote" "$REAL" --workdir "$state_dir/bisync" --max-lock 5m --resilient --recover + --max-delete 50 --exclude "/$marker_name" --log-file "$state_dir/sync.log" --log-level INFO) + [ "$resync" = yes ] && args+=(--resync) + [ "$force" = yes ] && args+=(--force) + + (umask 077; [ -f "$state_dir/sync.log" ] && mv -f "$state_dir/sync.log" "$state_dir/sync.log.1"; : > "$state_dir/sync.log") + touch "$run_dir/sync.started" + start="$(date +%s)" + say "sync: $remote <-> $REAL${why_resync:+ (--resync: $why_resync)}" + RCLONE_CONFIG="$conf" RCLONE_CACHE_DIR="$state_dir/cache" rclone "${args[@]}" >"$state_dir/sync.log" 2>&1 & + child=$! + # A stop (the timer's TERM, or Ctrl-C) asks bisync for its graceful shutdown (SIGINT), then + # after the grace period ends it with SIGTERM. + trap 'stopping=yes; kill -INT "$child" 2>/dev/null' TERM INT + wait "$child" 2>/dev/null; rc=$? + while alive "$child"; do + if [ "$stopping" = yes ]; then + i=0 + while alive "$child" && [ "$i" -lt $((grace * 2)) ]; do sleep 0.5; i=$((i + 1)); done + alive "$child" && kill -TERM "$child" 2>/dev/null + fi + wait "$child" 2>/dev/null; rc=$? + done + trap - TERM INT + secs=$(( $(date +%s) - start )) + + if [ "$rc" -eq 0 ] && [ "$stopping" = no ]; then result=ok + elif [ "$stopping" = yes ]; then result=interrupted + elif [ "$rc" -eq 7 ]; then result=critical + else result=failed; fi + # The first error that names a cause (not bisync's closing "aborted" lines), else the last one. + note="$(grep -E 'ERROR|Safety abort' "$state_dir/sync.log" 2>/dev/null | grep -v -E 'retryable|Bisync aborted' | head -n 1)" + [ -n "$note" ] || note="$(grep -E 'ERROR|Failed' "$state_dir/sync.log" 2>/dev/null | tail -n 1)" + note="$(printf '%s' "$note" | sed -E 's/^[0-9\/]+ [0-9:]+ //' | tr -d '\r' | cut -c1-300)" + [ "$result" = ok ] && note='' + (umask 077 + { + printf 'time=%s\n' "$(date -u -d "@$start" +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || date -u +%Y-%m-%dT%H:%M:%SZ)" + printf 'result=%s\nrc=%s\nresync=%s\nseconds=%s\nnote=%s\n' "$result" "$rc" "$resync" "$secs" "$note" + } > "$state_dir/last-sync.tmp" && mv -f "$state_dir/last-sync.tmp" "$state_dir/last-sync") + if [ "$result" = ok ]; then + touch -r "$run_dir/sync.started" "$state_dir/last-ok" + printf '%s\n' "$pair" > "$REAL/$marker_name" + printf '%s\n' "$pair" > "$state_dir/bisync-pair" + say "sync ok in ${secs}s${why_resync:+ (resync)}" + return 0 + fi + say "sync $result (rclone exit $rc)${note:+: $note}; log: $state_dir/sync.log" + case "$note" in + *'too many deletes'*) say "bisync's delete guard: restore the files, or if the deletion is meant, task cloud-folder:sync -- --force" ;; + *'Empty current Path'*) say 'one side is empty and bisync will not empty the other: if the folder was emptied by mistake, task cloud-folder:sync -- --resync brings the files back' ;; + esac + [ "$rc" -eq 7 ] && return 7 + return 1 +} + +# ----------------------------------------------------------------- timer ---- +start_timer() { + local secs p i + resolve + if [ "$MODE_USED" != sync ]; then say "timer: not started (mode $MODE_USED)"; return 0; fi + if [ "$CONF" != ok ]; then say 'timer: not started, no remote is configured yet (task cloud-folder:login starts it)'; return 0; fi + if ! secs="$(interval_seconds)"; then say "timer: not started, syncInterval '$sync_interval' is not a duration (15m, 900s, 1h, 0)"; return 1; fi + if [ "$secs" -eq 0 ]; then say 'timer: off (syncInterval 0): sync on demand with task cloud-folder:sync'; return 0; fi + p="$(pid_of "$run_dir/loop.pid")" + if ours "$p"; then say "timer: already running (pid $p, every $sync_interval)"; return 0; fi + prepare_state || { say "timer: not started, the state folder $state_dir is not writable"; return 1; } + if [ -f "$state_dir/timer.log" ] && [ "$(stat -c %s "$state_dir/timer.log" 2>/dev/null || echo 0)" -gt 1048576 ]; then + mv -f "$state_dir/timer.log" "$state_dir/timer.log.1" + fi + if command -v setsid >/dev/null 2>&1; then + (umask 077; setsid bash "$self" run >"$state_dir/timer.log" 2>&1 &) + else + (umask 077; nohup bash "$self" run >"$state_dir/timer.log" 2>&1 &) + fi + for i in 1 2 3 4 5 6 7 8 9 10; do + p="$(pid_of "$run_dir/loop.pid")" + ours "$p" && break + sleep 0.3 + done + if ours "$p"; then say "timer: started (pid $p, every $sync_interval; the first sync runs now); log: $state_dir/timer.log"; return 0; fi + say "timer: did not start; see $state_dir/timer.log" + return 1 +} + +wait_child() { # the loop's child: wait through signals until it is gone + wait "$1" 2>/dev/null + while alive "$1"; do wait "$1" 2>/dev/null; done +} + +run_loop() { + local secs child='' stopping=no + secs="$(interval_seconds)" || { say "timer: syncInterval '$sync_interval' is not a duration"; return 1; } + [ "$secs" -eq 0 ] && { say 'timer: syncInterval 0, no timer'; return 0; } + prepare_state || return 1 + if ! take_lock "$run_dir/loop.pid"; then say "timer: already running (pid $(pid_of "$run_dir/loop.pid"))"; return 0; fi + # shellcheck disable=SC2064 # the lock path is fixed now + trap "drop_lock '$run_dir/loop.pid'" EXIT + trap 'stopping=yes; [ -n "$child" ] && kill -TERM "$child" 2>/dev/null' TERM INT HUP + say "timer: started $(date -u +%Y-%m-%dT%H:%M:%SZ), pid $$, every $sync_interval" + while [ "$stopping" = no ]; do + bash "$self" sync --timer & + child=$!; wait_child "$child"; child='' + [ "$stopping" = yes ] && break + sleep "$secs" & + child=$!; wait_child "$child"; child='' + done + say "timer: stopped $(date -u +%Y-%m-%dT%H:%M:%SZ)" +} + +stop_timer() { + local p i max + max="${CLOUD_FOLDER_STOP_WAIT:-120}" + p="$(pid_of "$run_dir/loop.pid")" + if ! ours "$p"; then + [ -n "$p" ] && rm -f -- "$run_dir/loop.pid" + say 'timer: not running' + return 0 + fi + kill -TERM "$p" 2>/dev/null + i=0 + while ours "$p" && [ "$i" -lt $((max * 2)) ]; do sleep 0.5; i=$((i + 1)); done + if ours "$p"; then say "timer: still finishing a sync after ${max}s (pid $p); task cloud-folder:status shows when it ends"; return 1; fi + say 'timer: stopped' +} + +# shutdown_check [SECONDS]: exit 1 while a sync runs (after waiting) or local changes wait. +shutdown_check() { + local max="${1:-${CLOUD_FOLDER_SHUTDOWN_WAIT:-60}}" waited=0 p rc=0 + case "$max" in '' | *[!0-9]*) echo "cloud-folder: shutdown-check: SECONDS must be a number" >&2; return 2 ;; esac + resolve + if [ "$MODE_USED" != sync ]; then say "shutdown check: mode $MODE_USED, no sync in flight"; return 0; fi + p="$(pid_of "$run_dir/sync.pid")" + if ours "$p"; then + say "shutdown check: a sync is running (pid $p); waiting up to ${max}s for it to end" + while ours "$p" && [ "$waited" -lt "$max" ]; do sleep 1; waited=$((waited + 1)); done + fi + if ours "$p"; then + say "REFUSE: a sync is still running (pid $p) after ${max}s; a stop now would cut it short. Wait and run the check again (task cloud-folder:status), or stop with FORCE=1." + rc=1 + fi + read_record + local n + n="$(pending_count "$REAL")" + if [ "$CONF" = ok ] && [ "$n" -gt 0 ]; then + if [ -n "$LAST_TIME" ]; then + say "REFUSE: $n file(s) in $REAL changed since the last successful sync started; run task cloud-folder:sync first, or stop with FORCE=1." + else + say "REFUSE: $REAL has $n file(s) and has never been synced; run task cloud-folder:sync first, or stop with FORCE=1." + fi + rc=1 + elif [ "$CONF" != ok ] && [ "$n" -gt 0 ]; then + say "note: $REAL has $n file(s) and no remote is configured; they live only in this container." + fi + if [ -n "$LAST_RESULT" ] && [ "$LAST_RESULT" != ok ]; then say "note: the last sync ended $LAST_RESULT (task cloud-folder:status)."; fi + [ "$rc" -eq 0 ] && say "shutdown check: no sync running, nothing changed since the last sync${LAST_TIME:+ ($LAST_TIME)}: safe to stop" + return "$rc" +} + +# ----------------------------------------------------------------- login ---- +json_field() { # : the first "name": "string" value of rclone's pretty JSON + sed -n 's/^[[:space:]]*"'"$1"'":[[:space:]]*"\(.*\)",\{0,1\}[[:space:]]*$/\1/p' <<<"$2" | head -n 1 +} + +# section_write ...: replace the [cloud-folder] section of the config with these lines +# (other remotes kept). Written with shell builtins into a private file; nothing reaches argv. +section_write() { + local tmp="$conf.tmp.$$" line + (umask 077 + if [ -f "$conf" ]; then + awk -v sec="[$remote_name]" '/^[[:space:]]*\[/ { s = $0; gsub(/[[:space:]]/, "", s); skip = (s == sec) } !skip' "$conf" > "$tmp" || exit 1 + else + : > "$tmp" || exit 1 + fi) || return 1 + { + printf '[%s]\n' "$remote_name" + for line in "$@"; do printf '%s\n' "$line"; done + } >> "$tmp" || { rm -f -- "$tmp"; return 1; } + chmod 0600 "$tmp" && mv -f -- "$tmp" "$conf" +} + +# drive_lookup : run rclone's own non-interactive config for the remote, +# answering its questions, so it finds the drive (drive_id and drive_type) from the site. The +# answers are paths and choices, never a secret; rclone's replies are parsed, never shown whole. +drive_lookup() { + local secret="$1" out state name err result n i choice + shift + local -a values helps + out="$(rcl config update "$remote_name" "$@" --non-interactive 2>&1)" + for _ in 1 2 3 4 5 6 7 8; do + state="$(json_field State "$out")" + err="$(json_field Error "$out")" + name="$(json_field Name "$out")" + if [ -n "$err" ]; then + [ -n "$secret" ] && err="${err//"$secret"/[secret]}" + say "the drive lookup failed: $err" + return 1 + fi + if [ -z "$state" ]; then + if grep -q '"State"' <<<"$out"; then return 0; fi + out="$(printf '%s\n' "$out" | tail -n 3)" + [ -n "$secret" ] && out="${out//"$secret"/[secret]}" + say "the drive lookup gave no answer rclone understood: $out" + return 1 + fi + case "$name" in + config_driveid) + mapfile -t values < <(sed -n 's/^[[:space:]]*"Value":[[:space:]]*"\(.*\)",\{0,1\}[[:space:]]*$/\1/p' <<<"$out") + mapfile -t helps < <(sed -n 's/^[[:space:]]*"Help":[[:space:]]*"\(.*\)",\{0,1\}[[:space:]]*$/\1/p' <<<"$out") + n="${#values[@]}" + if [ "$n" -eq 0 ]; then say 'the site has no drive (document library) this sign-in can see'; return 1; fi + if [ "$n" -eq 1 ]; then result="${values[0]}"; say "drive: ${helps[0]:-${values[0]}}" + else + say "the site has $n drives (document libraries):" + for ((i = 0; i < n; i++)); do echo " $((i + 1))) ${helps[$i]:-} [${values[$i]}]"; done + if [ ! -t 0 ] && [ -z "${CLOUD_FOLDER_DRIVE_CHOICE:-}" ]; then + say 'choose one: set CLOUD_FOLDER_DRIVE_ID to its id (in brackets) and run login again' + return 1 + fi + choice="${CLOUD_FOLDER_DRIVE_CHOICE:-}" + if [ -z "$choice" ]; then printf 'cloud-folder: which one (1-%s)? ' "$n"; read -r choice || choice=''; fi + case "$choice" in '' | *[!0-9]*) say "not a number: '$choice'"; return 1 ;; esac + if [ "$choice" -lt 1 ] || [ "$choice" -gt "$n" ]; then say "not between 1 and $n: $choice"; return 1; fi + result="${values[$((choice - 1))]}" + fi ;; + config_drive_ok) result=true ;; + *) say "rclone asked a question this login does not know ($name); finish with: rclone config reconnect $remote_name: (RCLONE_CONFIG=$conf)"; return 1 ;; + esac + out="$(rcl config update "$remote_name" "$@" --non-interactive --continue --state "$state" --result "$result" 2>&1)" + done + say 'the drive lookup did not finish' + return 1 +} + +do_login() { + local v token secret='' auth lines drive_id="${CLOUD_FOLDER_DRIVE_ID:-}" ctype + local -a answers=() + command -v rclone >/dev/null 2>&1 || { say 'rclone is not on PATH: rebuild with the Feature in mode auto or sync'; return 3; } + prepare_state || { say "the state folder $state_dir is not writable"; return 3; } + case "$remote_type" in + sharepoint) + if [ -z "$drive_id" ] && [ -z "$site_url" ]; then say 'remoteType sharepoint needs siteUrl (the option, or CLOUD_FOLDER_SITE_URL), for example https://.sharepoint.com/sites/'; return 2; fi + ctype=url ;; + onedrive) ctype=onedrive ;; + *) say "remoteType '$remote_type' is not known (sharepoint or onedrive)"; return 2 ;; + esac + v="$(rclone version 2>/dev/null | head -n 1)"; v="${v#rclone }" + + if [ -n "${CLOUD_FOLDER_CLIENT_ID:-}" ]; then + # Client credentials (the cluster case; unverified, see NOTES): an app registration's id, its + # tenant, and its secret from a file the platform mounts or from the environment. + auth=client-credentials + if [ -z "${CLOUD_FOLDER_TENANT:-}" ]; then say 'client credentials need CLOUD_FOLDER_TENANT (the tenant id) too'; return 2; fi + if [ -n "${CLOUD_FOLDER_CLIENT_SECRET_FILE:-}" ]; then + [ -r "$CLOUD_FOLDER_CLIENT_SECRET_FILE" ] || { say "CLOUD_FOLDER_CLIENT_SECRET_FILE cannot be read: $CLOUD_FOLDER_CLIENT_SECRET_FILE"; return 2; } + secret="$(< "$CLOUD_FOLDER_CLIENT_SECRET_FILE")" + else + secret="${CLOUD_FOLDER_CLIENT_SECRET:-}" + fi + unset CLOUD_FOLDER_CLIENT_SECRET + secret="${secret//$'\r'/}"; secret="${secret%%$'\n'*}" + [ -n "$secret" ] || { say 'client credentials need a secret: CLOUD_FOLDER_CLIENT_SECRET_FILE (a file) or CLOUD_FOLDER_CLIENT_SECRET'; return 2; } + lines=("type = onedrive" "client_id = $CLOUD_FOLDER_CLIENT_ID" "client_secret = $secret" "tenant = $CLOUD_FOLDER_TENANT" "client_credentials = true" "drive_type = $drive_type") + else + auth=token + say "sign-in for remote '$remote_name' (remoteType $remote_type), rclone $v." + say "On a machine with a web browser, install the same rclone version ($v, https://downloads.rclone.org/$v/) and run:" + echo + echo ' rclone authorize "onedrive"' + echo + say 'sign in in the browser it opens; it prints a token between "Paste the following into your remote machine --->" and "<---End paste".' + [ -t 0 ] && printf 'cloud-folder: paste that token here and press Enter (it is not shown): ' + if [ -t 0 ]; then read -r -s token || token=''; echo; else read -r token || true; fi + token="${token//$'\r'/}" + token="${token#"${token%%[![:space:]]*}"}"; token="${token%"${token##*[![:space:]]}"}" + case "$token" in + '{'*'"access_token"'*'}') ;; + '') say 'no token pasted; nothing changed'; return 1 ;; + *) say 'that is not the token rclone authorize prints (one line of JSON with access_token); nothing changed. Check that both sides run the same rclone version.'; return 1 ;; + esac + secret="$token" + lines=("type = onedrive" "token = $token" "drive_type = $drive_type") + answers+=(config_refresh_token=false) + fi + [ -n "$drive_id" ] && lines+=("drive_id = $drive_id") + section_write "${lines[@]}" || { say "could not write $conf"; return 1; } + say "remote '$remote_name' written to $conf (mode 600, $auth)" + + if [ -z "$drive_id" ]; then + answers+=("config_type=$ctype") + [ "$ctype" = url ] && answers+=("config_site_url=$site_url") + if ! drive_lookup "$secret" "${answers[@]}"; then + say "the sign-in is kept, but the remote has no drive yet: fix the cause and run task cloud-folder:login again, or set CLOUD_FOLDER_DRIVE_ID" + return 1 + fi + fi + chmod 0600 "$conf" 2>/dev/null + secret=''; token='' + if rcl lsjson --stat --contimeout 10s --timeout 15s --low-level-retries 1 --retries 1 "$remote" >/dev/null 2>&1; then + say "signed in: $remote answers" + else + say "signed in, but $remote does not answer yet (a wrong remotePath? it must exist on the remote); task cloud-folder:status shows the error" + fi + start_timer + return 0 +} + +# ---------------------------------------------------------------- report ---- cmd="${1:-report}" [ $# -gt 0 ] && shift case "$cmd" in report) - assess + prepare_state verbose || true + resolve + [ "$MODE_USED" = sync ] && ensure_local "$REAL" >/dev/null 2>&1 + assess remote print_report + [ "$MODE_USED" = sync ] && start_timer exit 0 ;; status) json=no @@ -195,20 +825,33 @@ case "$cmd" in *) echo "cloud-folder: status: unknown argument '$a' (only --json)" >&2; exit 2 ;; esac done - assess + assess remote if [ "$json" = yes ]; then print_json; else print_report; fi verdict_ok && exit 0 exit 1 ;; version) - say "Feature $feature_version; mode option $mode_option; mount point $mount_point; host mount $host_mount" - if command -v rclone >/dev/null 2>&1; then say "rclone on PATH (not used by this version): $(rclone version 2>/dev/null | head -n 1)" - else say "rclone: not installed (this version needs none)"; fi + resolve + say "Feature $feature_version; mode option $mode_option (in use: $MODE_USED); mount point $mount_point; host mount $host_mount; state $state_dir" + if command -v rclone >/dev/null 2>&1; then say "rclone: $(rclone version 2>/dev/null | head -n 1)${rclone_version_option:+ (option rcloneVersion $rclone_version_option)}" + else say "rclone: not installed (mode host needs none)"; fi exit 0 ;; + login) do_login; exit $? ;; + sync) do_sync "$@"; exit $? ;; + start) start_timer; exit $? ;; + stop) stop_timer; exit $? ;; + run) run_loop; exit $? ;; + shutdown-check) + [ $# -le 1 ] || { echo "usage: cloud-folder-setup shutdown-check [SECONDS]" >&2; exit 2; } + shutdown_check "${1:-}"; exit $? ;; link) [ $# -eq 2 ] || { echo "usage: cloud-folder-setup link MOUNT_POINT HOST_MOUNT" >&2; exit 2; } link_mount_point "$1" "$2" exit $? ;; + folder) + [ $# -eq 1 ] || { echo "usage: cloud-folder-setup folder MOUNT_POINT" >&2; exit 2; } + make_folder "$1" + exit $? ;; *) - echo "usage: cloud-folder-setup [status [--json] | version | link MOUNT_POINT HOST_MOUNT]" >&2 + echo "usage: cloud-folder-setup [status [--json] | version | login | sync [--resync] [--force] | start | stop | shutdown-check [SECONDS] | link MOUNT_POINT HOST_MOUNT | folder MOUNT_POINT]" >&2 exit 2 ;; esac diff --git a/src/cloud-folder/dcf-persist.sh b/src/cloud-folder/dcf-persist.sh new file mode 100644 index 0000000..5dea79e --- /dev/null +++ b/src/cloud-folder/dcf-persist.sh @@ -0,0 +1,59 @@ +#!/usr/bin/env bash +# dcf-persist: make a tool's state directory usable by the current user and +# say where it lives. The one helper behind every -setup post-create hook +# of this collection (claude-code, git, gh, az), so they all behave the same: +# +# dcf-persist [--label ] ... +# +# For each : create it (with sudo when the parent is not writable), hand +# it to the current user when its top level is owned by someone else (named +# volumes and bind mounts arrive owned by root or by the host's uid), and +# print one line: label, path, what backs it, and whether it is writable. +# Backing is one of +# volume a named volume: survives rebuilds, deleted by a container-engine +# storage reset or a move to another engine; +# bind a folder from the host disk: survives all of that; +# image the container filesystem: gone at the next rebuild. +# Never fatal (exit 0 always): a problem is reported, and the rest of +# post-create goes on. Canonical copy: src/common/dcf-persist.sh; the other +# Features carry a verbatim copy, checked by `task lint`. +set -u + +label="" +if [ "${1:-}" = "--label" ]; then label="$2"; shift 2; fi +[ $# -gt 0 ] || { echo "usage: dcf-persist [--label ] ..." >&2; exit 0; } + +me="$(id -u):$(id -g)" +as_root() { # run a command as root: directly, or through passwordless sudo + if [ "$(id -u)" -eq 0 ]; then "$@"; elif command -v sudo >/dev/null 2>&1 && sudo -n true 2>/dev/null; then sudo -n "$@"; else return 1; fi +} + +backing_of() { # -> volume | bind | image + local src fstype + src="$(findmnt -no SOURCE --target "$1" 2>/dev/null || true)" + fstype="$(findmnt -no FSTYPE --target "$1" 2>/dev/null || true)" + # The container's own filesystem is the overlay root. A named volume shows + # the engine's volume store as its source (/var/lib/docker/volumes// + # _data, or the podman equivalent); any other mount came from outside. + case "$fstype" in overlay | "") echo image ;; *) case "$src" in *"/volumes/"*) echo volume ;; *) echo bind ;; esac ;; esac +} + +for dir in "$@"; do + name="${label:-$(basename "$dir")}" + if [ ! -d "$dir" ]; then + mkdir -p "$dir" 2>/dev/null || as_root mkdir -p "$dir" || { echo "$name: cannot create $dir" >&2; continue; } + fi + if [ "$(stat -c %u:%g "$dir" 2>/dev/null)" != "$me" ]; then + # Only the top level is checked, so a large folder is not walked on every + # start; -R because the contents came along with the same owner. + as_root chown -R "$me" "$dir" 2>/dev/null || true + fi + backing="$(backing_of "$dir")" + if [ -w "$dir" ]; then state=writable; else state="NOT writable by $(id -un)"; fi + echo "$name: $dir ($backing, $state)" + case "$backing" in + volume) echo "$name: a named volume is deleted by a container-engine storage reset; for the host disk, bind-mount the same target in devcontainer.json (see the Feature README)." ;; + image) echo "$name: inside the image, lost at the next rebuild; add the Feature's mount or set the directory variable to a mounted path." ;; + esac +done +exit 0 diff --git a/src/cloud-folder/devcontainer-feature.json b/src/cloud-folder/devcontainer-feature.json index a1e716c..682bbd5 100644 --- a/src/cloud-folder/devcontainer-feature.json +++ b/src/cloud-folder/devcontainer-feature.json @@ -1,8 +1,8 @@ { "id": "cloud-folder", - "version": "1.0.0", + "version": "1.1.0", "name": "Cloud folder", - "description": "One fixed path in the devcontainer (mountPoint, default /mnt/cloud-folder) for a shared document folder that must never go in git, such as a SharePoint or OneDrive folder. This version knows host mode only: the devcontainer-cli launcher bind-mounts the host folder named by DCF_HOST_CLOUD_FOLDER (a folder a sync client keeps on the host) at /mnt/cloud-folder-host, and mountPoint links to it; no rclone, no sign-in. A post-start hook (cloud-folder-setup) reports on every start which mode is in use and whether the folder is mounted, empty and writable, and cloud-folder:status exits non-zero when it is missing or unwritable, so a start-up check can stop. Plus the Feature Taskfile (version, status, setup).", + "description": "One fixed path in the devcontainer (mountPoint, default /mnt/cloud-folder) for a shared document folder that must never go in git, such as a SharePoint or OneDrive folder, filled by one of two modes. host: the devcontainer-cli launcher bind-mounts the host folder named by DCF_HOST_CLOUD_FOLDER (a folder a sync client keeps on the host) at /mnt/cloud-folder-host, and mountPoint links to it. sync: rclone (installed pinned and checksum-verified, only when absent) runs bisync between the remote and the folder, on demand and on a timer started at each start, with the token, config and sync state in the Feature's own volume; nothing privileged, so it works on a cluster. auto picks host when a host folder is mounted, else sync when a remote is signed in. A post-start hook (cloud-folder-setup) reports on every start; cloud-folder:status exits non-zero when the folder is missing, unwritable, never synced or its last sync failed, and cloud-folder:shutdown-check refuses a stop while a sync runs or local changes wait. Plus the Feature Taskfile (version, status, setup, login, sync, start, stop, shutdown-check).", "documentationURL": "https://github.com/nabil-becker/devcontainer-features/tree/main/src/cloud-folder", "licenseURL": "https://github.com/nabil-becker/devcontainer-features/blob/main/LICENSE", "options": { @@ -15,20 +15,72 @@ "type": "string", "enum": [ "auto", - "host" + "host", + "sync" ], "default": "auto", - "description": "Where the folder comes from. host: a bind mount the launcher adds from DCF_HOST_CLOUD_FOLDER (no rclone). auto: host when something is mounted at the mount point, else it reports that there is no source and never pretends. The rclone modes (sync, mount) come in a later version. Override per container with CLOUD_FOLDER_MODE in containerEnv, or per call on the task." + "description": "Where the folder comes from. host: a bind mount the launcher adds from DCF_HOST_CLOUD_FOLDER (no rclone). sync: rclone bisync between the remote and the folder, on a timer and on demand (mountPoint is then a plain folder, no link). auto: host when something is mounted at the mount point, else sync when a remote is signed in (task cloud-folder:login), else it reports that there is no source and never pretends. The mount mode comes in a later version. Override per container with CLOUD_FOLDER_MODE in containerEnv, or per call on the task." }, "mountPoint": { "type": "string", "default": "/mnt/cloud-folder", - "description": "Absolute path where the folder appears in the container. The launcher always mounts the host folder at /mnt/cloud-folder-host; when mountPoint is another path, the install makes it a symbolic link to that one, so this option is the only place to name the path. The install refuses a mountPoint that already exists in the image as a folder with content or as a file." + "description": "Absolute path where the folder appears in the container. In auto and host, the launcher always mounts the host folder at /mnt/cloud-folder-host and mountPoint is a symbolic link to it when it is another path (in auto, a sync fills the folder the link leads to). In sync, mountPoint is a plain folder, no link. The install refuses a mountPoint that already exists in the image as a folder with content (auto, host) or as a file." + }, + "remoteType": { + "type": "string", + "enum": [ + "sharepoint", + "onedrive" + ], + "default": "sharepoint", + "description": "sync: what the remote is. sharepoint: a document library of the SharePoint site siteUrl (the only kind the client-credentials sign-in of a cluster can use). onedrive: the signed-in person's own OneDrive. Both use rclone's onedrive backend. Override with CLOUD_FOLDER_REMOTE_TYPE." + }, + "driveType": { + "type": "string", + "enum": [ + "personal", + "business", + "documentLibrary" + ], + "default": "documentLibrary", + "description": "sync: rclone's drive_type for the remote, written at login; the drive lookup at login replaces it with what the drive reports. Override with CLOUD_FOLDER_DRIVE_TYPE." + }, + "siteUrl": { + "type": "string", + "default": "", + "description": "sync, remoteType sharepoint: the site's URL, for example https://.sharepoint.com/sites/; login finds the site's document library from it. Override with CLOUD_FOLDER_SITE_URL." + }, + "remotePath": { + "type": "string", + "default": "", + "description": "sync: the folder inside the drive (document library) to sync, for example Shared/Recordings; empty means the whole drive. It must exist on the remote. Override with CLOUD_FOLDER_REMOTE_PATH." + }, + "syncInterval": { + "type": "string", + "default": "15m", + "description": "sync: how often the timer runs a sync (a number of seconds, or with s, m or h: 900, 15m, 1h). 0: no timer, sync on demand only (task cloud-folder:sync). Override with CLOUD_FOLDER_SYNC_INTERVAL." + }, + "rcloneVersion": { + "type": "string", + "default": "1.75.2", + "description": "The rclone release to install for sync (with or without a leading v), or 'latest'. The default is pinned with its sha256 per architecture in this Feature; another version is verified against the release's SHA256SUMS. Not installed in mode host, nor when the image already has rclone (that one is kept)." } }, + "containerEnv": { + "RCLONE_CONFIG": "/var/lib/devcontainer-features/cloud-folder/rclone.conf" + }, + "mounts": [ + { + "source": "cloud-folder-${devcontainerId}", + "target": "/var/lib/devcontainer-features/cloud-folder", + "type": "volume" + } + ], "postStartCommand": "cloud-folder-setup", "installsAfter": [ "ghcr.io/nabil-becker/devcontainer-features/go-task", - "ghcr.io/devcontainers/features/common-utils" + "ghcr.io/devcontainers/features/common-utils", + "ghcr.io/devcontainers-extra/features/rclone", + "ghcr.io/devcontainer-community/devcontainer-features/rclone.org" ] } diff --git a/src/cloud-folder/install.sh b/src/cloud-folder/install.sh index 8f093d7..cafe1d2 100644 --- a/src/cloud-folder/install.sh +++ b/src/cloud-folder/install.sh @@ -1,27 +1,40 @@ #!/usr/bin/env bash -# Installs the cloud-folder Feature's pieces: cloud-folder-setup (the -# postStartCommand, also behind `task cloud-folder:status`) and the Feature -# Taskfile. Feature options arrive as upper-cased environment variables (see -# devcontainer-feature.json): MODE, MOUNTPOINT. +# Installs the cloud-folder Feature's pieces: rclone (for the sync mode; pinned and +# checksum-verified, only when absent), cloud-folder-setup (the postStartCommand, also behind +# every `task cloud-folder:*`), dcf-persist (the collection's state-folder helper) and the +# Feature Taskfile. Feature options arrive as upper-cased environment variables (see +# devcontainer-feature.json): MODE, MOUNTPOINT, REMOTETYPE, DRIVETYPE, SITEURL, REMOTEPATH, +# SYNCINTERVAL, RCLONEVERSION. # -# This version knows host mode only (a bind mount the launcher adds), so -# nothing is downloaded and there is nothing to verify. rclone, its config -# volume and the sync and mount modes come in a later version. +# Two layouts, chosen by the option mode: +# auto, host The launcher bind-mounts the host folder at one fixed path, the host mount +# (HOST_MOUNT below, the same constant as in both launcher twins). When the option +# mountPoint is another path, it becomes a symbolic link to the host mount, so a +# consumer names the path once. Neither path is created as a folder: when nothing +# is mounted the link dangles and the folder is absent. In auto, when no host +# folder is mounted and a remote is configured, the sync mode fills the folder the +# link leads to (cloud-folder-setup creates it at start). +# sync mountPoint is a plain folder of the image, filled by `rclone bisync`; no link. # -# The launcher bind-mounts the host folder at one fixed path, the host mount -# (HOST_MOUNT below, the same constant as in both launcher twins). When the -# option mountPoint is another path, it becomes a symbolic link to the host -# mount here, so a consumer names the path once (the option). Neither path is -# created as a folder: when nothing is mounted the link dangles and the folder -# is absent, so no one writes into a bare folder of the image by mistake; -# Docker creates the host mount when the launcher mounts onto it. The rclone -# modes of a later version fill mountPoint themselves and make no link. +# rclone: downloaded from the publisher (downloads.rclone.org) as the release zip, verified +# against the sha256 pinned in rclone-checksums.txt for that version and architecture or, for +# a version not listed there, against the release's SHA256SUMS; never piped into a shell. Not +# installed in mode host (it is not used there), and kept as it is when the image already has +# one (another Feature installed it first: installsAfter orders this one behind them). set -euo pipefail HOST_MOUNT=/mnt/cloud-folder-host +STATE_DIR=/var/lib/devcontainer-features/cloud-folder +DOWNLOADS=https://downloads.rclone.org MODE="${MODE:-auto}" MOUNTPOINT="${MOUNTPOINT:-/mnt/cloud-folder}" +REMOTETYPE="${REMOTETYPE:-sharepoint}" +DRIVETYPE="${DRIVETYPE:-documentLibrary}" +SITEURL="${SITEURL:-}" +REMOTEPATH="${REMOTEPATH:-}" +SYNCINTERVAL="${SYNCINTERVAL:-15m}" +RCLONEVERSION="${RCLONEVERSION:-1.75.2}" if [ "$(id -u)" -ne 0 ]; then echo "cloud-folder: install.sh must run as root (the devcontainer Feature lifecycle does this)." >&2 @@ -29,34 +42,127 @@ if [ "$(id -u)" -ne 0 ]; then fi case "$MODE" in - auto | host) ;; - *) echo "cloud-folder: option mode '$MODE' is not known to this version (auto or host)." >&2; exit 1 ;; + auto | host | sync) ;; + *) echo "cloud-folder: option mode '$MODE' is not known to this version (auto, host or sync)." >&2; exit 1 ;; esac case "$MOUNTPOINT" in /?*) ;; *) echo "cloud-folder: option mountPoint '$MOUNTPOINT' must be an absolute path." >&2; exit 1 ;; esac +case "$REMOTETYPE" in + sharepoint | onedrive) ;; + *) echo "cloud-folder: option remoteType '$REMOTETYPE' is not known (sharepoint or onedrive)." >&2; exit 1 ;; +esac +case "$DRIVETYPE" in + personal | business | documentLibrary) ;; + *) echo "cloud-folder: option driveType '$DRIVETYPE' is not known (personal, business or documentLibrary)." >&2; exit 1 ;; +esac +if [[ ! "$SYNCINTERVAL" =~ ^[0-9]+[smh]?$ ]]; then + echo "cloud-folder: option syncInterval '$SYNCINTERVAL' is not a duration (for example 15m, 900s, 1h, or 0 for on demand only)." >&2 + exit 1 +fi feature_dir="$(cd "$(dirname "$0")" && pwd)" version="$(sed -n 's/^[[:space:]]*"version":[[:space:]]*"\([^"]*\)".*/\1/p' "$feature_dir/devcontainer-feature.json" | head -n 1)" -# cloud-folder-setup reads its option defaults from options.env, so the hook -# and the tasks agree; CLOUD_FOLDER_* in the environment override them. +# ---------------------------------------------------------------- rclone ---- +install_rclone() { + local arch v asset sum tmp missing=() + command -v curl >/dev/null 2>&1 || missing+=(curl ca-certificates) + command -v unzip >/dev/null 2>&1 || missing+=(unzip) + if [ "${#missing[@]}" -gt 0 ]; then + if command -v apt-get >/dev/null 2>&1; then + apt-get update -qq + apt-get install -y -qq --no-install-recommends "${missing[@]}" + rm -rf /var/lib/apt/lists/* + else + echo "cloud-folder: missing ${missing[*]} and no apt-get to install them." >&2 + return 1 + fi + fi + case "$(uname -m)" in + x86_64 | amd64) arch=amd64 ;; + aarch64 | arm64) arch=arm64 ;; + *) echo "cloud-folder: rclone: unsupported architecture $(uname -m) (amd64 and arm64)" >&2; return 1 ;; + esac + + v="$RCLONEVERSION" + if [ "$v" = latest ]; then + # The publisher's one-line version file ("rclone v1.75.2"). + v="$(curl -fsSL "$DOWNLOADS/version.txt" | sed -n 's/^rclone v\{0,1\}\([0-9][0-9.]*\).*/\1/p' | head -n 1)" + echo "cloud-folder: rcloneVersion latest resolves to $v" + fi + v="${v#v}" + if [[ ! "$v" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]; then + echo "cloud-folder: could not resolve an rclone version from '$RCLONEVERSION'" >&2 + return 1 + fi + + asset="rclone-v${v}-linux-${arch}.zip" + tmp="$(mktemp -d)" + # shellcheck disable=SC2064 # expand now: the trap must remove this tmp + trap "rm -rf '$tmp'" RETURN + # 1. The digest pinned in this Feature: no trust in what the server says about itself. + sum="$(awk -v n="$asset" '$2 == n { print $1; exit }' "$feature_dir/rclone-checksums.txt" 2>/dev/null || true)" + if [ -z "$sum" ]; then + # 2. A version not pinned here: the release's SHA256SUMS from the publisher. + sum="$(curl -fsSL --retry 3 --retry-delay 2 "$DOWNLOADS/v${v}/SHA256SUMS" | awk -v n="$asset" '$2 == n { print $1; exit }' || true)" + [ -n "$sum" ] && echo "cloud-folder: rclone $v is not pinned in this Feature; verifying against the release's SHA256SUMS" + fi + if [[ ! "$sum" =~ ^[0-9a-f]{64}$ ]]; then + echo "cloud-folder: no sha256 for $asset (not in rclone-checksums.txt, not in the release's SHA256SUMS)." >&2 + return 1 + fi + echo "cloud-folder: downloading rclone v${v} (${asset})..." + curl -fsSL --retry 3 --retry-delay 2 -o "$tmp/$asset" "$DOWNLOADS/v${v}/$asset" + echo "${sum} $tmp/$asset" | sha256sum -c - + unzip -q -j -o "$tmp/$asset" "rclone-v${v}-linux-${arch}/rclone" -d "$tmp/x" + install -m 0755 "$tmp/x/rclone" /usr/local/bin/rclone + /usr/local/bin/rclone version | head -n 1 +} + +if [ "$MODE" = host ]; then + echo "cloud-folder: mode host: rclone is not installed (host mode does not use it)" +elif found="$(command -v rclone 2>/dev/null)"; then + echo "cloud-folder: keeping the rclone already in the image ($found, $("$found" version 2>/dev/null | head -n 1)); rcloneVersion '$RCLONEVERSION' is ignored." +else + install_rclone +fi + +# ------------------------------------------------------- ecosystem pieces ---- +# cloud-folder-setup reads its option defaults from options.env, so the hook and the tasks +# agree; CLOUD_FOLDER_* in the environment override them. share=/usr/local/share/cloud-folder-feature mkdir -p "$share" { echo "# Written by the cloud-folder Feature's install.sh from its options; sourced by cloud-folder-setup." printf 'MODE=%q\n' "$MODE" printf 'MOUNTPOINT=%q\n' "$MOUNTPOINT" + printf 'REMOTETYPE=%q\n' "$REMOTETYPE" + printf 'DRIVETYPE=%q\n' "$DRIVETYPE" + printf 'SITEURL=%q\n' "$SITEURL" + printf 'REMOTEPATH=%q\n' "$REMOTEPATH" + printf 'SYNCINTERVAL=%q\n' "$SYNCINTERVAL" + printf 'RCLONEVERSION=%q\n' "$RCLONEVERSION" printf 'FEATURE_VERSION=%q\n' "${version:-unknown}" printf 'HOST_MOUNT=%q\n' "$HOST_MOUNT" + printf 'STATE_DIR=%q\n' "$STATE_DIR" } > "$share/options.env" chmod 0644 "$share/options.env" install -m 0755 "$feature_dir/cloud-folder-setup.sh" /usr/local/bin/cloud-folder-setup +install -m 0755 "$feature_dir/dcf-persist.sh" /usr/local/bin/dcf-persist -# Host mode (all this version knows): mountPoint leads to the host mount. A refusal (a folder with -# content, or a file, already at mountPoint) fails the build with the reason. -/usr/local/bin/cloud-folder-setup link "$MOUNTPOINT" "$HOST_MOUNT" +# The state folder is the Feature's volume (devcontainer-feature.json), mounted at start, not +# at build: nothing is written into it here. cloud-folder-setup hands it to the remote user. +mkdir -p "$STATE_DIR" + +# The mount point. A refusal (a folder with content, or a file, where the link must go; a file +# where the folder must go) fails the build with the reason. +if [ "$MODE" = sync ]; then + /usr/local/bin/cloud-folder-setup folder "$MOUNTPOINT" +else + /usr/local/bin/cloud-folder-setup link "$MOUNTPOINT" "$HOST_MOUNT" +fi echo "cloud-folder: Feature ${version:-unknown}, mode $MODE, mount point $MOUNTPOINT, host mount $HOST_MOUNT" diff --git a/src/cloud-folder/rclone-checksums.txt b/src/cloud-folder/rclone-checksums.txt new file mode 100644 index 0000000..fe72511 --- /dev/null +++ b/src/cloud-folder/rclone-checksums.txt @@ -0,0 +1,7 @@ +# sha256 of rclone release zips, " ", one per line, checked on +# 2026-10-27 against both downloads.rclone.org/v/SHA256SUMS and the GitHub release's +# SHA256SUMS (they agree). install.sh checks this file first; a version not listed here is +# verified against the publisher's SHA256SUMS at build. Add a version with the one-liner in +# NOTES.md ("Version and upgrade"). +349ac8fba6ff65d6247043f1750cdcb518ec5d500ef91463a10d37c0ccdf3702 rclone-v1.75.2-linux-amd64.zip +7e1e8d69654941b7b7df84ee74c5f7fb09cce7d5496947fb7bf16b55ff427d10 rclone-v1.75.2-linux-arm64.zip diff --git a/src/cloud-folder/taskfile.yml b/src/cloud-folder/taskfile.yml index 5052696..3828643 100644 --- a/src/cloud-folder/taskfile.yml +++ b/src/cloud-folder/taskfile.yml @@ -1,23 +1,68 @@ # Tasks contributed by the cloud-folder Feature (namespace: cloud-folder). -# CLOUD_FOLDER_MODE and CLOUD_FOLDER_MOUNT_POINT, from the command line, a .env file or the -# environment (in that order, as Task resolves them), override the Feature options for one call. +# CLOUD_FOLDER_MODE, CLOUD_FOLDER_MOUNT_POINT, CLOUD_FOLDER_REMOTE_TYPE, CLOUD_FOLDER_DRIVE_TYPE, +# CLOUD_FOLDER_SITE_URL, CLOUD_FOLDER_REMOTE_PATH and CLOUD_FOLDER_SYNC_INTERVAL, from the command +# line, a .env file or the environment (in that order, as Task resolves them), override the +# Feature options for one call. Secrets (CLOUD_FOLDER_CLIENT_SECRET*) are read from the +# environment only, never put on a command line. version: '3' tasks: version: dir: '{{.USER_WORKING_DIR}}' - desc: 'Show the cloud-folder Feature version, the mode option and the mount point in use, and whether rclone is on PATH (this version uses none).' + desc: 'Show the cloud-folder Feature version, the mode option and the mode in use, the mount point, the state folder, and the rclone version.' + vars: &cf + CF: '{{if .CLOUD_FOLDER_MODE}}CLOUD_FOLDER_MODE={{shellQuote .CLOUD_FOLDER_MODE}} {{end}}{{if .CLOUD_FOLDER_MOUNT_POINT}}CLOUD_FOLDER_MOUNT_POINT={{shellQuote .CLOUD_FOLDER_MOUNT_POINT}} {{end}}{{if .CLOUD_FOLDER_REMOTE_TYPE}}CLOUD_FOLDER_REMOTE_TYPE={{shellQuote .CLOUD_FOLDER_REMOTE_TYPE}} {{end}}{{if .CLOUD_FOLDER_DRIVE_TYPE}}CLOUD_FOLDER_DRIVE_TYPE={{shellQuote .CLOUD_FOLDER_DRIVE_TYPE}} {{end}}{{if .CLOUD_FOLDER_SITE_URL}}CLOUD_FOLDER_SITE_URL={{shellQuote .CLOUD_FOLDER_SITE_URL}} {{end}}{{if .CLOUD_FOLDER_REMOTE_PATH}}CLOUD_FOLDER_REMOTE_PATH={{shellQuote .CLOUD_FOLDER_REMOTE_PATH}} {{end}}{{if .CLOUD_FOLDER_SYNC_INTERVAL}}CLOUD_FOLDER_SYNC_INTERVAL={{shellQuote .CLOUD_FOLDER_SYNC_INTERVAL}} {{end}}' cmds: - - '{{if .CLOUD_FOLDER_MODE}}CLOUD_FOLDER_MODE={{shellQuote .CLOUD_FOLDER_MODE}} {{end}}{{if .CLOUD_FOLDER_MOUNT_POINT}}CLOUD_FOLDER_MOUNT_POINT={{shellQuote .CLOUD_FOLDER_MOUNT_POINT}} {{end}}cloud-folder-setup version' + - '{{.CF}}cloud-folder-setup version' status: dir: '{{.USER_WORKING_DIR}}' - desc: 'Report on the cloud folder: the mode in use and why, whether the mount point is mounted, empty and writable, and what to do when it is not. Exit 0 when it is mounted and writable (an empty folder is reported, not failed), 1 when it is missing, not mounted, not writable or the mode is unknown, so a start-up check can stop. `-- --json` prints one JSON object for agents. Writes nothing into the folder.' + desc: 'Report on the cloud folder: the mode in use and why; in host mode whether the mount point is mounted, empty and writable; in sync mode the remote (configured, reachable; no token is printed), the last sync (time, result), whether a sync or the timer runs, and the files changed since the last sync. Exit 0 when the folder is usable (an empty folder is reported, not failed), 1 when it is missing, not mounted, not writable, has no remote, was never synced, its last sync failed, or the mode is unknown, so a start-up check can stop. `-- --json` prints one JSON object for agents. Writes nothing into the folder.' + vars: *cf cmds: - - '{{if .CLOUD_FOLDER_MODE}}CLOUD_FOLDER_MODE={{shellQuote .CLOUD_FOLDER_MODE}} {{end}}{{if .CLOUD_FOLDER_MOUNT_POINT}}CLOUD_FOLDER_MOUNT_POINT={{shellQuote .CLOUD_FOLDER_MOUNT_POINT}} {{end}}cloud-folder-setup status {{.CLI_ARGS}}' + - '{{.CF}}cloud-folder-setup status {{.CLI_ARGS}}' setup: dir: '{{.USER_WORKING_DIR}}' - desc: 'Re-run the post-start hook: the same report as status, but never fatal (exit 0).' + desc: 'Re-run the post-start hook: hand the state volume (and in sync mode the folder) to this user, the same report as status, and in sync mode start the timer if it is not running. Never fatal (exit 0).' + vars: *cf cmds: - - '{{if .CLOUD_FOLDER_MODE}}CLOUD_FOLDER_MODE={{shellQuote .CLOUD_FOLDER_MODE}} {{end}}{{if .CLOUD_FOLDER_MOUNT_POINT}}CLOUD_FOLDER_MOUNT_POINT={{shellQuote .CLOUD_FOLDER_MOUNT_POINT}} {{end}}cloud-folder-setup' + - '{{.CF}}cloud-folder-setup' + + login: + dir: '{{.USER_WORKING_DIR}}' + desc: 'Sign in for sync mode. Interactive: run `rclone authorize "onedrive"` (the same rclone version, which this prints) on a machine with a browser and paste the token it prints; it is read without echo and written only into the Feature volume''s rclone.conf (mode 600). Then finds the drive from siteUrl (remoteType sharepoint) or the person''s OneDrive, checks the remote answers and starts the timer. With CLOUD_FOLDER_CLIENT_ID, CLOUD_FOLDER_TENANT and CLOUD_FOLDER_CLIENT_SECRET_FILE (or CLOUD_FOLDER_CLIENT_SECRET) in the environment it uses client credentials instead, without a prompt (unverified, see the Feature NOTES). CLOUD_FOLDER_DRIVE_ID skips the drive lookup.' + interactive: true + vars: *cf + cmds: + - '{{.CF}}cloud-folder-setup login' + + sync: + dir: '{{.USER_WORKING_DIR}}' + desc: 'Run one rclone bisync between the remote and the folder now (the first sync of a folder is a --resync on its own). `-- --resync` forces one; `-- --force` lets a sync through bisync''s delete guard (more than half of one side deleted) when the deletion is meant. Exit 0 ok, 1 failed, 7 bisync wants a --resync, 3 cannot sync here (mode, sign-in), 4 another sync runs.' + vars: *cf + cmds: + - '{{.CF}}cloud-folder-setup sync {{.CLI_ARGS}}' + + start: + dir: '{{.USER_WORKING_DIR}}' + desc: 'Start the sync timer (one loop, every syncInterval; the post-start hook does this in sync mode). Idempotent: a running timer is left as it is. syncInterval 0 starts none.' + vars: *cf + cmds: + - '{{.CF}}cloud-folder-setup start' + + stop: + dir: '{{.USER_WORKING_DIR}}' + desc: 'Stop the sync timer; a sync that is running gets bisync''s graceful shutdown first (up to about 100 s). Exit 1 when it is still finishing after CLOUD_FOLDER_STOP_WAIT seconds (default 120).' + vars: *cf + cmds: + - '{{.CF}}cloud-folder-setup stop' + + shutdown-check: + dir: '{{.USER_WORKING_DIR}}' + desc: 'For a repository''s shutdown:check (bootstrap/SHUTDOWN.md): exit 1 while a sync runs (after waiting up to WAIT seconds, default 60) or while files in the folder changed since the last successful sync; exit 0 otherwise, and in host mode.' + vars: + CF: '{{if .CLOUD_FOLDER_MODE}}CLOUD_FOLDER_MODE={{shellQuote .CLOUD_FOLDER_MODE}} {{end}}{{if .CLOUD_FOLDER_MOUNT_POINT}}CLOUD_FOLDER_MOUNT_POINT={{shellQuote .CLOUD_FOLDER_MOUNT_POINT}} {{end}}' + WAIT: '{{.WAIT | default ""}}' + cmds: + - '{{.CF}}cloud-folder-setup shutdown-check {{.WAIT}}' diff --git a/test/cloud-folder/host_only.sh b/test/cloud-folder/host_only.sh new file mode 100644 index 0000000..a47d255 --- /dev/null +++ b/test/cloud-folder/host_only.sh @@ -0,0 +1,13 @@ +#!/bin/bash +# Scenario 'host_only' (see scenarios.json): mode host installs no rclone (host mode does not use it) +# and makes the link as before. +set -e +# shellcheck disable=SC1091 # provided by the devcontainer CLI at test time +source dev-container-features-test-lib + +check "no rclone in mode host" bash -c "! command -v rclone" +check "the mount point links to the host mount" bash -c "[ \"\$(readlink /mnt/cloud-folder)\" = /mnt/cloud-folder-host ]" +check "status fails as not-mounted" bash -c "cloud-folder-setup status | grep -F 'not usable (not-mounted)'" +check "the shutdown check has nothing to wait for" bash -c "cloud-folder-setup shutdown-check 0 | grep -F 'no sync in flight'" + +reportResults diff --git a/test/cloud-folder/pinned_version.sh b/test/cloud-folder/pinned_version.sh new file mode 100644 index 0000000..47ed955 --- /dev/null +++ b/test/cloud-folder/pinned_version.sh @@ -0,0 +1,14 @@ +#!/bin/bash +# Scenario 'pinned_version' (see scenarios.json): rcloneVersion=1.75.1, a version not pinned in the +# Feature's rclone-checksums.txt, so the install verified it against the release's SHA256SUMS. +set -e +# shellcheck disable=SC1091 # provided by the devcontainer CLI at test time +source dev-container-features-test-lib + +check "rclone 1.75.1 installed" bash -c "rclone version | head -n 1 | grep -Fx 'rclone v1.75.1'" +check "the option is baked" grep -qx 'RCLONEVERSION=1.75.1' /usr/local/share/cloud-folder-feature/options.env +# shellcheck disable=SC2016 # expansion is meant to happen inside the inner bash +check "runs as non-root user" bash -c '[ "$(id -u)" -ne 0 ] && rclone version >/dev/null' +check "version names it" bash -c "cloud-folder-setup version | grep -F 'rclone: rclone v1.75.1'" + +reportResults diff --git a/test/cloud-folder/scenarios.json b/test/cloud-folder/scenarios.json index ba47e8e..b5cbf27 100644 --- a/test/cloud-folder/scenarios.json +++ b/test/cloud-folder/scenarios.json @@ -42,5 +42,36 @@ } }, "remoteUser": "vscode" + }, + "pinned_version": { + "image": "mcr.microsoft.com/devcontainers/base:debian", + "features": { + "cloud-folder": { + "rcloneVersion": "1.75.1" + } + }, + "remoteUser": "vscode" + }, + "host_only": { + "image": "mcr.microsoft.com/devcontainers/base:debian", + "features": { + "cloud-folder": { + "mode": "host" + } + }, + "remoteUser": "vscode" + }, + "sync_local": { + "image": "mcr.microsoft.com/devcontainers/base:debian", + "features": { + "go-task": {}, + "cloud-folder": { + "mode": "sync", + "mountPoint": "/mnt/shared-docs", + "remotePath": "sub", + "syncInterval": "0" + } + }, + "remoteUser": "vscode" } } diff --git a/test/cloud-folder/sync_local.sh b/test/cloud-folder/sync_local.sh new file mode 100644 index 0000000..de353af --- /dev/null +++ b/test/cloud-folder/sync_local.sh @@ -0,0 +1,68 @@ +#!/bin/bash +# Scenario 'sync_local' (see scenarios.json): mode sync with the real rclone, as vscode, mountPoint +# /mnt/shared-docs (a plain folder, no link), remotePath sub, syncInterval 0. No SharePoint: the +# remote cloud-folder is an rclone alias of a local folder, written into the Feature volume's config +# with a token line that must never show up anywhere else. Covers the first sync (a resync), a +# file each way, the status, the shutdown check refusing pending changes, the timer started once +# and stopped, and the token never in a process's arguments, a log or the setup output. +set -e +# shellcheck disable=SC1091 # provided by the devcontainer CLI at test time +source dev-container-features-test-lib + +state=/var/lib/devcontainer-features/cloud-folder +marker='TOKENMARKER-not-a-real-token' +out=/tmp/cf-output +: > "$out" + +check "mountPoint is a folder, not a link" bash -c "test -d /mnt/shared-docs && ! test -L /mnt/shared-docs" +check "the post-start hook runs (again) and hands the folders to the user" bash -c "cloud-folder-setup >> $out 2>&1 && [ \"\$(stat -c %U $state)\" = vscode ]" +check "the sync folder is the user's before any sign-in" bash -c "[ \"\$(stat -c %U /mnt/shared-docs)\" = vscode ] && test -w /mnt/shared-docs" + +mkdir -p /tmp/cf-remote/sub +echo from-remote > /tmp/cf-remote/sub/remote.txt +(umask 077; printf '[cloud-folder]\ntype = alias\nremote = /tmp/cf-remote\ntoken = {"access_token":"%s"}\n' "$marker" > "$RCLONE_CONFIG") + +check "configured, never synced: status fails" bash -c "! cloud-folder-setup status >> $out 2>&1" +check "and says never-synced" bash -c "cloud-folder-setup status | tee -a $out | grep -F 'not usable (never-synced)'" +check "the first sync is a resync and succeeds" bash -c "cloud-folder-setup sync 2>&1 | tee -a $out | grep -F '(--resync: first sync of /mnt/shared-docs)'" +check "the remote file arrived" grep -qx from-remote /mnt/shared-docs/remote.txt +check "status passes" bash -c "cloud-folder-setup status >> $out 2>&1" +check "status --json says ok and reachable" bash -c "cloud-folder-setup status --json | tee -a $out | grep -F '\"remote_reachable\":true' | grep -F '\"verdict\":\"ok\"'" +sleep 1.1 +echo from-local > /mnt/shared-docs/local.txt +check "the shutdown check refuses a pending change" bash -c "! cloud-folder-setup shutdown-check 0 >> $out 2>&1" +check "a second sync (no resync) sends it up" bash -c "cloud-folder-setup sync 2>&1 | tee -a $out | grep -E '^cloud-folder: sync ok in [0-9]+s\$'" +check "the local file is on the remote" grep -qx from-local /tmp/cf-remote/sub/local.txt +check "the marker is not on the remote" bash -c "! test -e /tmp/cf-remote/sub/.cloud-folder-bisync" +check "the shutdown check passes" bash -c "cloud-folder-setup shutdown-check 0 >> $out 2>&1" +check "the config is mode 600" bash -c "[ \"\$(stat -c %a \$RCLONE_CONFIG)\" = 600 ]" +check "bisync's listings live in the volume" bash -c "ls $state/bisync | grep -q '\\.lst\$'" + +check "the timer starts" bash -c "CLOUD_FOLDER_SYNC_INTERVAL=2s cloud-folder-setup start | tee -a $out | grep -F 'timer: started'" +check "a second start leaves it" bash -c "CLOUD_FOLDER_SYNC_INTERVAL=2s cloud-folder-setup start | tee -a $out | grep -F 'timer: already running'" +# The loop's forked subshells keep its arguments, and pgrep -f also matches a shell whose command +# line quotes the pattern, so count only processes whose arguments end in "cloud-folder-setup run" +# and whose parent is not such a process, and check that one is the PID file's. +one_loop() { + local p pp n=0 loop + loop="$(cat "$state/run/loop.pid")" + for p in $(pgrep -f 'cloud-folder-setup run$'); do + pp="$(ps -o ppid= -p "$p" | tr -d ' ')" + ps -o args= -p "$pp" 2>/dev/null | grep -q 'cloud-folder-setup run$' && continue + n=$((n + 1)); [ "$p" = "$loop" ] || return 1 + done + [ "$n" = 1 ] +} +check "one loop" one_loop +# Sample every process's arguments while the timer syncs. +for _ in $(seq 1 30); do ps -eo args >> /tmp/cf-ps; sleep 0.2; done +check "the timer synced" bash -c "grep -F 'sync ok' $state/timer.log" +check "the timer stops" bash -c "cloud-folder-setup stop | tee -a $out | grep -F 'timer: stopped'" +check "no loop left" bash -c "! pgrep -f 'cloud-folder-setup run$'" + +check "the token is in no process's arguments" bash -c "! grep -F '$marker' /tmp/cf-ps" +check "the token is in no output" bash -c "! grep -F '$marker' $out" +check "the token is in no file of the volume but the config" bash -c "[ -z \"\$(grep -rlF '$marker' $state | grep -v '/rclone.conf\$')\" ]" +check "task cloud-folder:sync works" task -t /usr/local/share/go-task/includes.d/cloud-folder.yml sync + +reportResults diff --git a/test/cloud-folder/test.sh b/test/cloud-folder/test.sh index f907c43..853df88 100644 --- a/test/cloud-folder/test.sh +++ b/test/cloud-folder/test.sh @@ -9,7 +9,7 @@ source dev-container-features-test-lib check "cloud-folder-setup on PATH" command -v cloud-folder-setup check "cloud-folder-setup is root-owned" bash -c "[ \"\$(stat -c %U \"\$(command -v cloud-folder-setup)\")\" = root ]" check "options baked for cloud-folder-setup" bash -c "grep -qx 'MODE=auto' /usr/local/share/cloud-folder-feature/options.env && grep -qx 'MOUNTPOINT=/mnt/cloud-folder' /usr/local/share/cloud-folder-feature/options.env" -check "the Feature version is baked" grep -qx 'FEATURE_VERSION=1.0.0' /usr/local/share/cloud-folder-feature/options.env +check "the Feature version is baked" grep -qx 'FEATURE_VERSION=1.1.0' /usr/local/share/cloud-folder-feature/options.env check "the host mount path is baked" grep -qx 'HOST_MOUNT=/mnt/cloud-folder-host' /usr/local/share/cloud-folder-feature/options.env check "the mount point is a link to the host mount" bash -c "[ \"\$(readlink /mnt/cloud-folder)\" = /mnt/cloud-folder-host ]" check "the host mount is not created by the image" bash -c "! test -e /mnt/cloud-folder-host" @@ -19,8 +19,14 @@ check "status fails with no source" bash -c "! cloud-folder-setup status" check "status says no source" bash -c "cloud-folder-setup status | grep -F 'not usable (no-source)'" check "status shows the link" bash -c "cloud-folder-setup status | grep -F 'mount point /mnt/cloud-folder -> /mnt/cloud-folder-host: missing'" check "status --json is JSON with verdict no-source" bash -c "cloud-folder-setup status --json | grep -F '\"verdict\":\"no-source\"'" -check "CLOUD_FOLDER_MODE=sync is refused by this version" bash -c "CLOUD_FOLDER_MODE=sync cloud-folder-setup status | grep -F 'not in this version'" -check "rclone is not installed by this version" bash -c "! command -v rclone" +check "CLOUD_FOLDER_MODE=mount is refused by this version" bash -c "CLOUD_FOLDER_MODE=mount cloud-folder-setup status | grep -F 'not in this version'" +check "CLOUD_FOLDER_MODE=sync with no remote says to sign in" bash -c "CLOUD_FOLDER_MODE=sync cloud-folder-setup status | grep -F 'not usable (not-configured)'" +check "rclone installed at the pinned default" bash -c "rclone version | head -n 1 | grep -Fx 'rclone v1.75.2'" +check "rclone is root-owned in /usr/local/bin" bash -c "[ \"\$(stat -c %U /usr/local/bin/rclone)\" = root ]" +check "RCLONE_CONFIG points into the Feature volume" bash -c "[ \"\$RCLONE_CONFIG\" = /var/lib/devcontainer-features/cloud-folder/rclone.conf ]" +check "the state folder is a volume" bash -c "awk '\$5 == \"/var/lib/devcontainer-features/cloud-folder\"' /proc/self/mountinfo | grep -q ." +check "no timer without a remote" bash -c "! test -e /var/lib/devcontainer-features/cloud-folder/run/loop.pid" +check "the shutdown check passes with no sync mode" cloud-folder-setup shutdown-check 0 check "taskfile registered" test -f /usr/local/share/go-task/includes.d/cloud-folder.yml diff --git a/test/cloud-folder/unit/status_test.sh b/test/cloud-folder/unit/status_test.sh index f645313..cb5d8e5 100644 --- a/test/cloud-folder/unit/status_test.sh +++ b/test/cloud-folder/unit/status_test.sh @@ -6,8 +6,8 @@ # link from mountPoint to the host mount (`link`) and status through it; no # source (auto, nothing mounted), mounted and not empty, mounted and empty, mounted and not # writable (skipped as root, which may write anywhere), a bare folder that exists but is not -# mounted, mode host with nothing mounted (and the DCF_HOST_CLOUD_FOLDER hint), modes this -# version does not know, a path with a space and a trailing slash, a mounted file, --json, the +# mounted, mode host with nothing mounted (and the DCF_HOST_CLOUD_FOLDER hint), the mode this +# version does not know (mount), a path with a space and a trailing slash, a mounted file, --json, the # post-start report never failing, nothing written into the folder, and the Feature Taskfile # handing CLOUD_FOLDER_* and CLI_ARGS to the script. Part of `task lint` # (`task test:cloud-folder-unit`). SETUP_SH points the tests at another copy of the script. @@ -48,7 +48,7 @@ run() { local -a envs=() while [ "$1" != -- ]; do envs+=("$1"); shift; done shift - out="$(env -i PATH="$PATH" HOME="$T" DCF_CLOUD_FOLDER_MOUNTINFO="$T/mountinfo" DCF_CLOUD_FOLDER_HOST_MOUNT="$T/cloud folder" "${envs[@]}" bash "$setup_sh" "$@" 2>&1)" + out="$(env -i PATH="$PATH" HOME="$T" DCF_CLOUD_FOLDER_STATE_DIR="$T/state" DCF_CLOUD_FOLDER_MOUNTINFO="$T/mountinfo" DCF_CLOUD_FOLDER_HOST_MOUNT="$T/cloud folder" "${envs[@]}" bash "$setup_sh" "$@" 2>&1)" rc=$? } field() { python3 -c 'import json, sys; v = json.loads(sys.stdin.read())[sys.argv[1]]; print(json.dumps(v) if not isinstance(v, str) else v)' "$1" <<<"$out"; } @@ -125,13 +125,11 @@ run CLOUD_FOLDER_MOUNT_POINT="$T/file" -- status check 'a mounted file: exit 1' 1 "$rc" has 'a mounted file: not a folder' 'not usable (not-a-folder)' "$out" -# --- modes this version does not know +# --- a mode this version does not know (mount comes later; sync is tested in sync_test.sh) mount_table "$mp" -for m in sync mount; do - run CLOUD_FOLDER_MOUNT_POINT="$mp" CLOUD_FOLDER_MODE="$m" -- status - check "mode $m: exit 1" 1 "$rc" - has "mode $m: not in this version" "mode $m is not in this version of the Feature" "$out" -done +run CLOUD_FOLDER_MOUNT_POINT="$mp" CLOUD_FOLDER_MODE=mount -- status +check "mode mount: exit 1" 1 "$rc" +has "mode mount: not in this version" "mode mount is not in this version of the Feature" "$out" run CLOUD_FOLDER_MOUNT_POINT="$mp" CLOUD_FOLDER_MODE=bogus -- status check 'an unknown mode: exit 1' 1 "$rc" has 'an unknown mode: says so' "mode 'bogus' is unknown" "$out" @@ -225,14 +223,14 @@ if command -v task >/dev/null 2>&1; then printf '#!/bin/sh\nexec bash %q "$@"\n' "$setup_sh" > "$T/bin/cloud-folder-setup" chmod +x "$T/bin/cloud-folder-setup" mount_table "$mp" - out="$(cd "$T" && env -i PATH="$T/bin:$PATH" HOME="$T" DCF_CLOUD_FOLDER_MOUNTINFO="$T/mountinfo" task -s -t "$taskfile" status CLOUD_FOLDER_MOUNT_POINT="$mp" -- --json 2>&1)" + out="$(cd "$T" && env -i PATH="$T/bin:$PATH" HOME="$T" DCF_CLOUD_FOLDER_STATE_DIR="$T/state" DCF_CLOUD_FOLDER_MOUNTINFO="$T/mountinfo" task -s -t "$taskfile" status CLOUD_FOLDER_MOUNT_POINT="$mp" -- --json 2>&1)" rc=$? check 'task status: a command-line CLOUD_FOLDER_MOUNT_POINT and -- --json reach the script' 'ok' "$(field verdict 2>/dev/null)" - out="$(cd "$T" && env -i PATH="$T/bin:$PATH" HOME="$T" DCF_CLOUD_FOLDER_MOUNTINFO="$T/mountinfo" CLOUD_FOLDER_MOUNT_POINT="$T/none" task -s -t "$taskfile" status 2>&1)" + out="$(cd "$T" && env -i PATH="$T/bin:$PATH" HOME="$T" DCF_CLOUD_FOLDER_STATE_DIR="$T/state" DCF_CLOUD_FOLDER_MOUNTINFO="$T/mountinfo" CLOUD_FOLDER_MOUNT_POINT="$T/none" task -s -t "$taskfile" status 2>&1)" rc=$? check 'task status: the environment is read too, and a failure fails the task' yes "$([ "$rc" -ne 0 ] && echo yes || echo no)" has 'task status: the environment value is used' "mount point $T/none: missing" "$out" - out="$(cd "$T" && env -i PATH="$T/bin:$PATH" HOME="$T" DCF_CLOUD_FOLDER_MOUNTINFO="$T/mountinfo" task -s -t "$taskfile" setup CLOUD_FOLDER_MOUNT_POINT="$T/none" 2>&1)" + out="$(cd "$T" && env -i PATH="$T/bin:$PATH" HOME="$T" DCF_CLOUD_FOLDER_STATE_DIR="$T/state" DCF_CLOUD_FOLDER_MOUNTINFO="$T/mountinfo" task -s -t "$taskfile" setup CLOUD_FOLDER_MOUNT_POINT="$T/none" 2>&1)" check 'task setup: never fails' 0 "$?" else echo "skip - the Taskfile checks: task is not on PATH" diff --git a/test/cloud-folder/unit/sync_test.sh b/test/cloud-folder/unit/sync_test.sh new file mode 100644 index 0000000..2691f99 --- /dev/null +++ b/test/cloud-folder/unit/sync_test.sh @@ -0,0 +1,406 @@ +#!/usr/bin/env bash +# Docker-free tests of cloud-folder-setup's sync mode (src/cloud-folder/cloud-folder-setup.sh) +# with a fake rclone on PATH: it records every call's arguments, answers `version`, `lsjson +# --stat` and `config update` (rclone's non-interactive config protocol), and plays bisync +# between two temp folders (the remote is an alias remote pointing at one of them), with a +# chosen delay, exit code and log line. The real rclone runs in the Docker scenarios +# (test/cloud-folder/sync_local.sh). Covered: auto choosing sync once a remote is signed in; +# login (a pasted token read from stdin and written only into the config, mode 600; a wrong +# paste refused; siteUrl required for sharepoint; one drive taken, several listed and chosen; +# client credentials from a secret file); the first sync a --resync with the flags that matter +# (--max-lock, --max-delete 50, --resilient, --recover, the workdir in the volume); later syncs +# without it; a new folder (the marker gone) resynced; a failure and bisync's delete guard +# reported and failing the status; exit 7; status and --json; the files changed since the last +# sync; a second sync refused while one runs; the shutdown check refusing a running sync and +# pending changes, waiting for a sync to end, and passing in host mode; the timer started once +# (a second start leaves it), syncing on its own, off with syncInterval 0, stopped cleanly, and +# stopping a running sync gracefully; the token never in a process's arguments, the logs, the +# record or any output; the Feature Taskfile passing -- --resync and WAIT. Part of `task lint` +# (`task test:cloud-folder-unit`). SETUP_SH points the tests at another copy of the script. +set -u +exec /dev/null; done + pkill -f "$T/bin/rclone" 2>/dev/null + sleep 0.3 + chmod -R u+w "$T" 2>/dev/null; rm -rf "$T" +} +trap cleanup EXIT + +pass=0 fail=0 +check() { # + if [ "$2" = "$3" ]; then pass=$((pass + 1)); echo "ok - $1" + else fail=$((fail + 1)); echo "FAIL - $1 (expected '$2', got '$3')"; fi +} +has() { # + if grep -qF -- "$2" <<<"$3"; then pass=$((pass + 1)); echo "ok - $1" + else fail=$((fail + 1)); echo "FAIL - $1 (no '$2' in: $3)"; fi +} +hasnt() { # + if grep -qF -- "$2" <<<"$3"; then fail=$((fail + 1)); echo "FAIL - $1 ('$2' found in: $3)" + else pass=$((pass + 1)); echo "ok - $1"; fi +} +yesno() { if "$@"; then echo yes; else echo no; fi; } + +SECRET='SECRETTOKENMARKER-7f3a' +CSECRET='CLIENTSECRETMARKER-91c2' +state="$T/state" +mp="$T/cloud folder" +remote_dir="$T/remote" +mkdir -p "$T/bin" "$remote_dir/sub" +mount_table() { echo '22 1 8:1 / / rw,relatime - ext4 /dev/sda1 rw' > "$T/mountinfo"; } +mount_table + +# The fake rclone. FAKE_SLEEP seconds inside bisync (TERM ends it with 143; INT is what bisync's +# graceful shutdown listens for, but a background job of a script starts with INT ignored, so the +# setup script follows it with TERM), FAKE_RC its exit code, FAKE_ERROR a log line, +# FAKE_DRIVES the number of drives `config update` offers, FAKE_UNREACHABLE=1 fails lsjson. +cat > "$T/bin/rclone" <<'FAKE' +#!/usr/bin/env bash +printf '%s\n' "$*" >> "$FAKE_CALLS" +printf '%s\n' "$*" >> "$FAKE_CALLS.all" +case "$1" in + version) echo 'rclone v1.75.2'; exit 0 ;; + lsjson) [ "${FAKE_UNREACHABLE:-}" = 1 ] && { echo 'NOTICE: Failed to lsjson: directory not found' >&2; exit 3; }; echo '{}'; exit 0 ;; + config) + case " $* " in + *' --continue '*) printf '{\n\t"State": "",\n\t"Option": null,\n\t"Error": "",\n\t"Result": ""\n}\n'; exit 0 ;; + esac + if [ "${FAKE_DRIVES:-1}" -gt 1 ]; then + printf '{\n\t"State": "driveid_final",\n\t"Option": {\n\t\t"Name": "config_driveid",\n\t\t"Examples": [\n' + for i in $(seq 1 "$FAKE_DRIVES"); do printf '\t\t\t{\n\t\t\t\t"Value": "drive-%s",\n\t\t\t\t"Help": "Library %s (documentLibrary)"\n\t\t\t},\n' "$i" "$i"; done + printf '\t\t]\n\t},\n\t"Error": "",\n\t"Result": ""\n}\n' + else + printf '{\n\t"State": "driveid_final_end",\n\t"Option": {\n\t\t"Name": "config_drive_ok",\n\t\t"Default": true\n\t},\n\t"Error": "",\n\t"Result": ""\n}\n' + fi + exit 0 ;; + bisync) + p1="$2"; p2="$3"; shift 3 + resync=no workdir='' log='' + while [ $# -gt 0 ]; do + case "$1" in + --resync) resync=yes ;; + --workdir) workdir="$2"; shift ;; + --log-file) log="$2"; shift ;; + esac + shift + done + base="$(sed -n 's/^remote = //p' "$RCLONE_CONFIG" | head -n 1)" + src="$base/${p1#cloud-folder:}" + echo $$ > "$FAKE_PIDFILE" + trap 'echo "NOTICE: terminated" >> "$log"; exit 143' TERM + if [ -n "${FAKE_SLEEP:-}" ]; then + i=0; while [ "$i" -lt "$((FAKE_SLEEP * 10))" ]; do sleep 0.1; i=$((i + 1)); done + fi + if [ "$resync" = no ] && [ ! -f "$workdir/listing" ]; then + echo '2026/10/27 10:00:00 ERROR : Bisync critical error: cannot find prior Path1 or Path2 listings' >> "$log"; exit 7 + fi + if [ -n "${FAKE_RC:-}" ] && [ "$FAKE_RC" != 0 ]; then + echo "2026/10/27 10:00:00 ERROR : ${FAKE_ERROR:-something failed}" >> "$log"; exit "$FAKE_RC" + fi + cp -a -u "$src"/. "$p2"/ 2>/dev/null + (cd "$p2" && for f in * ; do [ -e "$f" ] && cp -a -u "$f" "$src"/; done) + : > "$workdir/listing" + echo '2026/10/27 10:00:00 INFO : Bisync successful' >> "$log" + exit 0 ;; +esac +echo "fake rclone: unexpected call: $*" >&2 +exit 2 +FAKE +chmod +x "$T/bin/rclone" +export FAKE_CALLS="$T/calls" FAKE_PIDFILE="$T/fake.pid" +: > "$FAKE_CALLS" + +ALL_OUT="$T/all-output" +: > "$ALL_OUT" +# run [VAR=value ...] --