Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@ diskpush ./data/ prod:/data/ -- --checksum # your own rsync flags
```

- **Local → server**, **server → local**, and **server → server** directly.
- **Sync a whole directory, or just what you picked.** Tick files and folders
in the pane, or `--only NAME` from the CLI.
- **Archive metadata by default.** Permissions, timestamps, symlinks.
- **Resumable by default.** An interrupted transfer keeps its partial data.
- **Skips unchanged files.** Re-running a job moves almost nothing.
Expand Down
3 changes: 3 additions & 0 deletions apps/cli/src/commands/transfer-helpers.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,10 @@ export {
planTransfer,
runPlan,
runToCompletion,
writeSelectionList,
describeSelection,
type ExecutionPlan,
type SelectionList,
} from '@diskpush/rsync-core'

import { runToCompletion, type ExecutionPlan } from '@diskpush/rsync-core'
Expand Down
63 changes: 62 additions & 1 deletion apps/cli/src/commands/transfer.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,15 +7,18 @@ import {
intersectCapabilities,
planTransfer,
runPlan,
writeSelectionList,
describeSelection,
summarizeChangesFrom,
type ExecutionPlan,
type SelectionList,
} from './transfer-helpers.js'
import type { Change, RsyncOptions } from '@diskpush/schemas'
import { summarizeChanges, topologyOf } from '@diskpush/schemas'
import { EXIT } from '../exit-codes.js'
import { estimateRemaining, formatBytes, formatDuration, formatRate, pluralize, table } from '../format.js'
import { failure, type Output } from '../output.js'
import { flagValue, hasFlag, type ParsedArgv } from '../parse-argv.js'
import { flagValue, flagValues, hasFlag, type ParsedArgv } from '../parse-argv.js'
import { detectLocalCapabilities, optionsFromFlags, resolveEndpoint } from '../resolve.js'
import type { RsyncCapabilities } from '@diskpush/rsync-core'

Expand Down Expand Up @@ -61,6 +64,61 @@ export async function runTransfer(

if (alias.deleteMode !== 'off') options.deleteMode = alias.deleteMode

/*
* `--only NAME` transfers just the entries named, rather than everything in
* the source directory — the thing an SFTP client makes trivial and a bare
* rsync does not.
*
* The names go to rsync as a NUL-separated `--files-from` list, which is not
* bounded by the command-line length limit and can express any name a
* filesystem allows. `writeSelectionList` refuses `..` and absolute paths:
* a selection is a choice among what the source directory holds, so a name
* is the only thing it can be.
*/
const only = flagValues(parsed, '--only')
let selection: SelectionList | null = null
if (only.length > 0) {
if (options.filesFrom) {
return failure(output, '--only and --files-from both choose what to send; use one.', EXIT.usage)
}
try {
selection = writeSelectionList(only)
} catch (error) {
return failure(output, (error as Error).message, EXIT.usage)
}
options.filesFrom = selection.path
options.from0 = true
}

try {
return await runResolvedTransfer(
command,
parsed,
store,
output,
alias,
sourceInput,
destinationInput,
options,
only,
)
} finally {
// rsync reads the list at startup, but it is not gone until the run is.
selection?.cleanup()
}
}

async function runResolvedTransfer(
command: string,
parsed: ParsedArgv,
store: DiskPushStore,
output: Output,
alias: (typeof TRANSFER_ALIASES)[string],
sourceInput: string,
destinationInput: string,
options: RsyncOptions,
only: readonly string[],
): Promise<number> {
const source = await resolveEndpoint(store, sourceInput)
const destination = await resolveEndpoint(store, destinationInput)
const topology = topologyOf(source.endpoint, destination.endpoint)
Expand Down Expand Up @@ -180,6 +238,9 @@ export async function runTransfer(
output.line(`DiskPush: ${alias.label} ${describe(sourceInput)} -> ${describe(destinationInput)}`)
output.line(`Source: ${sourceInput}`)
output.line(`Destination: ${destinationInput}`)
// Named before the transfer runs, for the same reason a mirror shows its
// delete list: what is about to move is worth stating.
if (only.length > 0) output.line(`Only: ${describeSelection(only)}`)
if (topology === 'remote-to-remote') {
output.line('')
output.line(`Direct path: ${source.endpoint.type === 'ssh' ? source.endpoint.host : '?'} -> ${destination.endpoint.type === 'ssh' ? destination.endpoint.host : '?'}`)
Expand Down
1 change: 1 addition & 0 deletions apps/cli/src/parse-argv.ts
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ export const VALUE_FLAGS = new Set([
'--exclude-from',
'--include-from',
'--files-from',
'--only',
'--bwlimit',
'--max-size',
'--min-size',
Expand Down
41 changes: 38 additions & 3 deletions apps/desktop/electron/main/services/transfers.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@ import { randomUUID } from 'node:crypto'
import type { WebContents } from 'electron'
import {
intersectCapabilities,
writeSelectionList,
type SelectionList,
parseRsyncCapabilities,
planTransfer,
runPlan,
Expand Down Expand Up @@ -102,11 +104,32 @@ function optionsFrom(input: TransferOptions): RsyncOptions {
})
}

async function buildPlan(request: TransferRequest, overrides: Partial<RsyncOptions> = {}): Promise<ExecutionPlan> {
/**
* Turns the renderer's selection into a `--files-from` list.
*
* Returns null when nothing is selected, which means the whole directory —
* the behaviour the two-pane view has always had. The caller removes the list
* once rsync has exited.
*/
function selectionFor(request: TransferRequest): SelectionList | null {
return request.selection.length > 0 ? writeSelectionList(request.selection) : null
}

async function buildPlan(
request: TransferRequest,
overrides: Partial<RsyncOptions> = {},
selection: SelectionList | null = null,
): Promise<ExecutionPlan> {
const source = await resolveEndpoint(request.source)
const destination = await resolveEndpoint(request.destination)
const capabilities = await capabilitiesFor([source.connectionId, destination.connectionId])
const options = { ...optionsFrom(request.options), ...overrides }
if (selection) {
options.filesFrom = selection.path
// NUL-separated: a newline is legal in a filename, so a newline-separated
// list cannot express every name a directory can hold.
options.from0 = true
}

const isServerToServer = source.endpoint.type === 'ssh' && destination.endpoint.type === 'ssh'
const sourceConnection = source.connectionId ? await resolveConnection(source.connectionId) : null
Expand Down Expand Up @@ -142,7 +165,15 @@ export type PreviewResult = {

/** The dry run behind Preview Changes and behind every mirror. */
export async function previewTransfer(request: TransferRequest): Promise<PreviewResult> {
const plan = await buildPlan(request, { dryRun: true })
const selection = selectionFor(request)
try {
return await previewWithPlan(await buildPlan(request, { dryRun: true }, selection))
} finally {
selection?.cleanup()
}
}

async function previewWithPlan(plan: ExecutionPlan): Promise<PreviewResult> {
const result = await runToCompletion(plan)
return {
changes: result.changes,
Expand All @@ -159,7 +190,10 @@ export async function previewTransfer(request: TransferRequest): Promise<Preview
export type StartedJob = { jobId: string; command: string; control: string | null; warnings: string[] }

export async function startTransfer(request: TransferRequest, sender: WebContents): Promise<StartedJob> {
const plan = await buildPlan(request)
// rsync reads the list at startup, but the run owns it until it exits: the
// cleanup below is in the event loop's `finally`, not this function's.
const selection = selectionFor(request)
const plan = await buildPlan(request, {}, selection)
const jobId = randomUUID()
const db = await store()

Expand Down Expand Up @@ -221,6 +255,7 @@ export async function startTransfer(request: TransferRequest, sender: WebContent
}
}
running.delete(jobId)
selection?.cleanup()
})()

return { jobId, command: plan.display, control: plan.controlDisplay ?? null, warnings: plan.warnings }
Expand Down
44 changes: 27 additions & 17 deletions apps/desktop/electron/shared/contract.ts
Original file line number Diff line number Diff line change
Expand Up @@ -113,12 +113,39 @@ export const TransferOptionsSchema = z.object({
})
export type TransferOptions = z.infer<typeof TransferOptionsSchema>

/**
* A single entry name inside a directory — never a path.
*
* Every mutating operation takes a directory plus one of these and joins them
* in the main process, so the renderer cannot walk out of the folder it is
* showing. `..`, a separator or a NUL would each be a way to do exactly that.
*/
export const EntryNameSchema = z
.string()
.min(1)
.max(255)
.refine((name) => !name.includes('/') && !name.includes('\\') && !name.includes('\0'), {
message: 'A name cannot contain a path separator.',
})
.refine((name) => name !== '.' && name !== '..', { message: 'That name is reserved.' })
.refine((name) => name.trim() === name, { message: 'A name cannot begin or end with a space.' })

export const TransferRequestSchema = z.object({
source: EndpointRefSchema,
destination: EndpointRefSchema,
options: TransferOptionsSchema,
/** Only meaningful for a delete-enabled job, and only after a preview. */
deletesConfirmed: z.boolean().default(false),
/**
* Send only these entries, rather than everything in the source directory.
*
* Entry names, not paths: each is one item the source pane is showing. The
* main process turns them into a `--files-from` list; the renderer never
* builds a path, so a selection cannot address anything the pane is not
* already looking at. Empty means the whole directory, which is what the
* two-pane view has always done.
*/
selection: z.array(EntryNameSchema).max(10_000).default([]),
})
export type TransferRequest = z.infer<typeof TransferRequestSchema>

Expand All @@ -135,23 +162,6 @@ export const RenameRequestSchema = z.object({
to: PathSchema,
})

/**
* A single entry name inside a directory — never a path.
*
* Every mutating operation takes a directory plus one of these and joins them
* in the main process, so the renderer cannot walk out of the folder it is
* showing. `..`, a separator or a NUL would each be a way to do exactly that.
*/
export const EntryNameSchema = z
.string()
.min(1)
.max(255)
.refine((name) => !name.includes('/') && !name.includes('\\') && !name.includes('\0'), {
message: 'A name cannot contain a path separator.',
})
.refine((name) => name !== '.' && name !== '..', { message: 'That name is reserved.' })
.refine((name) => name.trim() === name, { message: 'A name cannot begin or end with a space.' })

/** Create a directory or an empty file: `name` inside `directory`. */
export const CreateEntryRequestSchema = z.object({
connectionId: ConnectionIdSchema.optional(),
Expand Down
5 changes: 5 additions & 0 deletions apps/desktop/src/app/page.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -191,6 +191,10 @@ export default function Workspace() {
destination: refFor(destination, withTrailingSlash(destination.path)),
options: { deleteMode: mirror ? ('delay' as const) : ('off' as const) },
deletesConfirmed: false,
// Ticked entries in the source pane mean "just these", the way an SFTP
// client behaves. Nothing ticked keeps the old meaning: the whole
// directory.
selection: [...source.selected],
}),
[source, destination, mirror],
)
Expand Down Expand Up @@ -409,6 +413,7 @@ export default function Workspace() {
/>

<TransferRail
selectedCount={source.selected.size}
direction={direction}
mirror={mirror}
busy={job !== null && !job.finished}
Expand Down
18 changes: 16 additions & 2 deletions apps/desktop/src/components/transfer-rail.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -27,12 +27,15 @@ function Direction({
label,
armed,
busy,
selectedCount,
onClick,
}: {
side: 'left' | 'right'
label: string
armed: boolean
busy: boolean
/** Ticked entries in the source pane. 0 means the whole directory. */
selectedCount: number
onClick: () => void
}) {
const Arrow = side === 'right' ? ArrowRight : ArrowLeft
Expand Down Expand Up @@ -60,11 +63,17 @@ function Direction({
armed ? 'text-primary-foreground/70' : 'text-faint',
)}
>
Sync to
{armed && selectedCount > 0 ? `Send ${selectedCount}` : 'Sync to'}
</span>
<span className="w-full truncate text-center text-[11px] font-semibold leading-none">{label}</span>
</TooltipTrigger>
<TooltipContent>Copy into {label}, overwriting what differs</TooltipContent>
<TooltipContent>
{/* Only on the armed button: the count is the *source* pane's, and
the other button would make the other pane the source. */}
{armed && selectedCount > 0
? `Copy the ${selectedCount} selected ${selectedCount === 1 ? 'entry' : 'entries'} into ${label}`
: `Copy into ${label}, overwriting what differs`}
</TooltipContent>
</Tooltip>
)
}
Expand Down Expand Up @@ -118,6 +127,7 @@ export function TransferRail({
direction,
mirror,
busy,
selectedCount,
leftLabel,
rightLabel,
onDirection,
Expand All @@ -128,6 +138,8 @@ export function TransferRail({
direction: 'ltr' | 'rtl'
mirror: boolean
busy: boolean
/** Ticked entries in the source pane. 0 means the whole directory. */
selectedCount: number
leftLabel: string
rightLabel: string
onDirection: (direction: 'ltr' | 'rtl') => void
Expand All @@ -143,6 +155,7 @@ export function TransferRail({
<div className="flex w-[116px] shrink-0 flex-col items-center justify-center gap-2 px-3">
<Direction
side="right"
selectedCount={selectedCount}
label={rightLabel}
armed={direction === 'ltr'}
busy={busy}
Expand All @@ -153,6 +166,7 @@ export function TransferRail({
/>
<Direction
side="left"
selectedCount={selectedCount}
label={leftLabel}
armed={direction === 'rtl'}
busy={busy}
Expand Down
31 changes: 31 additions & 0 deletions docs/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -171,6 +171,7 @@ job would need a background daemon, which does not exist yet.

| Option | Effect |
| --- | --- |
| `--only NAME` | Send only this entry from the source directory, rather than all of it. Repeatable. |
| `-n`, `--dry-run` | Show the change set; transfer nothing. |
| `--print-args` | Print the exact rsync command and exit. |
| `--preset NAME` | `fast-sync`, `exact-mirror`, `maximum-metadata`, `slow-wan`, `verify-everything`. |
Expand All @@ -197,6 +198,36 @@ job would need a background daemon, which does not exist yet.
flag it is a boolean, so that `diskpush sync --compress ./a/ ./b/` cannot
mistake an endpoint for its value.

## Sending only some of a directory

`--only` transfers just the entries named, rather than everything in the
source — the thing an SFTP client makes trivial and a bare rsync does not.

```bash
diskpush ./site/ prod:/var/www/ --only index.html --only assets
diskpush pull prod:/var/log/ ./logs/ --only 'app.log'
```

Each name is one entry **inside the source directory**. A folder comes across
whole, with its contents. Names with spaces are fine. `..` and absolute paths
are refused: a selection is a choice among what the source holds, so a name is
the only thing it can be.

The names reach rsync as a NUL-separated `--files-from` list, which has two
consequences worth knowing:

- **`--files-from` turns recursion off, and `--archive` does not turn it back
on.** DiskPush restates `--recursive` so a selected folder arrives with its
contents rather than as an empty directory. Verified against rsync 3.4.1 —
it is not what the flag summary suggests.
- **`--delete` stays scoped to the selection.** `diskpush mirror SRC DST --only
cache` removes destination files inside `cache/` and leaves the rest of the
destination alone.

In the desktop app this is the pane selection: tick entries in the source pane
and the transfer button changes from *Sync to web-01* to *Send 2 → web-01*.
Nothing ticked means the whole directory, as before.

## Pass-through

```bash
Expand Down
Loading
Loading