diff --git a/docs/tutorials/command_line_client.md b/docs/tutorials/command_line_client.md index 5cc7da890..d3a6758d5 100644 --- a/docs/tutorials/command_line_client.md +++ b/docs/tutorials/command_line_client.md @@ -108,17 +108,31 @@ synapse get [-h] [-q queryString] [-v VERSION] [-r] [--followLink] [--limitSearc ### `manifest` -Generate manifest for uploading directory tree to Synapse. +Generate a TSV manifest describing the files in a local directory tree for +later upload with `synapse sync`. + +!!! warning "Folders are created in Synapse during manifest generation" + While walking the directory tree, `synapse manifest` **also creates the + corresponding folder structure under `--parent-id` in Synapse**, including + any empty subfolders. Only the files themselves are deferred to the + subsequent `synapse sync` call. That is why `--parent-id` is required even + when you only want to generate a manifest: each row in the manifest needs + to reference the Synapse ID of its (already-created) parent folder. + +To upload a subset of files, run `synapse manifest` first, delete the rows for +files you do not want to upload from the generated manifest, then run +`synapse sync `. The empty folders created in Synapse during manifest +generation will remain. ```bash synapse manifest [-h] --parent-id syn123 [--manifest-file OUTPUT] PATH ``` -| Name | Type | Description | Default | -|-------------------|------------|----------------------------------------------------------------|---------| -| `PATH` | Positional | A path to a file or folder whose manifest will be generated. | | -| `--parent-id` | Named | Synapse ID of project or folder where to upload data. | | -| `--manifest-file` | Named | A TSV output file path where the generated manifest is stored. | stdout | +| Name | Type | Description | Default | +|-------------------|------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------| +| `PATH` | Positional | Local path to the directory whose contents will be described in the manifest. The directory tree is walked recursively. | | +| `--parent-id` | Named | Synapse ID of the project or folder that will be the parent of the uploaded tree. Required even when only generating a manifest, because the matching folder structure (including empty folders) is created in Synapse under this parent as the manifest is generated. | | +| `--manifest-file` | Named | Path where the generated TSV manifest will be written (for example `./manifest.tsv`). If omitted, the manifest is printed to stdout; pass an explicit path if you intend to edit it before running `synapse sync`. | stdout | ### `sync` diff --git a/docs/tutorials/python/upload_data_in_bulk.md b/docs/tutorials/python/upload_data_in_bulk.md index b017163b1..f3725e2ea 100644 --- a/docs/tutorials/python/upload_data_in_bulk.md +++ b/docs/tutorials/python/upload_data_in_bulk.md @@ -67,6 +67,20 @@ It walks our local directory and produces a CSV manifest that maps each file to the correct parent folder in Synapse (creating folders as needed). The output is ready to hand directly to `sync_to_synapse`. +!!! warning "Folders are created in Synapse during manifest generation" + While walking the directory tree, `generate_sync_manifest` **creates the + corresponding folder structure on the project/folder it is called on**, + including any empty subfolders . Only the files themselves are deferred + to the later `sync_to_synapse` call. This is why the call still needs a + target Synapse Project/Folder even when you only want to produce a manifest + — each row in the manifest references the Synapse ID of its (already-created) + parent folder. + + To upload a subset of files, generate the manifest first, delete the rows + for files you do not want to upload from `PATH_TO_MANIFEST_FILE`, then run + `sync_to_synapse`. The empty folders created during manifest generation + will remain in Synapse. + ```python --8<-- "docs/tutorials/python/tutorial_scripts/upload_data_in_bulk.py:generate_manifest" ``` diff --git a/synapseclient/__main__.py b/synapseclient/__main__.py index fa462bb4c..34338112d 100644 --- a/synapseclient/__main__.py +++ b/synapseclient/__main__.py @@ -1006,13 +1006,26 @@ def build_parser(): parser_get.set_defaults(func=get) parser_manifest = subparsers.add_parser( - "manifest", help="Generate manifest for uploading directory tree to Synapse." + "manifest", + help="Generate manifest for uploading directory tree to Synapse.", + description=( + "Generate a TSV manifest describing the files in a local directory tree " + "for later upload with `synapse sync`. " + "IMPORTANT: this command will also create the corresponding folder structure " + "in Synapse while walking the directory under --parent-id " + "(including any empty folders); only the files are deferred to " + "the subsequent `synapse sync` call. " + "To upload a subset of files, run `synapse manifest` first, delete the rows " + "for files you do not want to upload from the generated manifest, then run " + "`synapse sync `." + ), ) parser_manifest.add_argument( "path", metavar="PATH", type=str, - help="A path to a file or folder whose manifest will be generated.", + help="Local path to the directory whose contents will be described in the " + "manifest. The directory tree is walked recursively.", ) parser_manifest.add_argument( "--parent-id", @@ -1020,13 +1033,18 @@ def build_parser(): type=str, required=True, dest="parentid", - help="Synapse ID of project or folder where to upload data.", + help="Synapse ID of the project or folder that will be the parent of the " + "uploaded tree. Required even when only generating a manifest, because the " + "matching folder structure (including empty folders) is created in Synapse " + "under this parent as the manifest is generated, so that each row in the " + "manifest can reference the Synapse ID of its parent folder.", ) parser_manifest.add_argument( "--manifest-file", metavar="OUTPUT", - help="A TSV output file path where the generated manifest is stored. " - "(default: stdout)", + help="Path where the generated TSV manifest will be written (for example " + "`./manifest.tsv`). If omitted, the manifest is printed to stdout; pass an " + "explicit path if you intend to edit it before running `synapse sync`.", ) parser_manifest.set_defaults(func=manifest)