Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 20 additions & 6 deletions docs/tutorials/command_line_client.md
Original file line number Diff line number Diff line change
Expand Up @@ -108,17 +108,31 @@ synapse get [-h] [-q queryString] [-v VERSION] [-r] [--followLink] [--limitSearc

### `manifest`

Generate manifest for uploading directory tree to Synapse.
Generate a TSV manifest describing the files in a local directory tree for
later upload with `synapse sync`.

!!! warning "Folders are created in Synapse during manifest generation"
While walking the directory tree, `synapse manifest` **also creates the
corresponding folder structure under `--parent-id` in Synapse**, including
any empty subfolders. Only the files themselves are deferred to the
subsequent `synapse sync` call. That is why `--parent-id` is required even
when you only want to generate a manifest: each row in the manifest needs
to reference the Synapse ID of its (already-created) parent folder.

To upload a subset of files, run `synapse manifest` first, delete the rows for
files you do not want to upload from the generated manifest, then run
`synapse sync <manifest>`. The empty folders created in Synapse during manifest
generation will remain.

```bash
synapse manifest [-h] --parent-id syn123 [--manifest-file OUTPUT] PATH
```

| Name | Type | Description | Default |
|-------------------|------------|----------------------------------------------------------------|---------|
| `PATH` | Positional | A path to a file or folder whose manifest will be generated. | |
| `--parent-id` | Named | Synapse ID of project or folder where to upload data. | |
| `--manifest-file` | Named | A TSV output file path where the generated manifest is stored. | stdout |
| Name | Type | Description | Default |
|-------------------|------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------|
| `PATH` | Positional | Local path to the directory whose contents will be described in the manifest. The directory tree is walked recursively. | |
| `--parent-id` | Named | Synapse ID of the project or folder that will be the parent of the uploaded tree. Required even when only generating a manifest, because the matching folder structure (including empty folders) is created in Synapse under this parent as the manifest is generated. | |
| `--manifest-file` | Named | Path where the generated TSV manifest will be written (for example `./manifest.tsv`). If omitted, the manifest is printed to stdout; pass an explicit path if you intend to edit it before running `synapse sync`. | stdout |


### `sync`
Expand Down
14 changes: 14 additions & 0 deletions docs/tutorials/python/upload_data_in_bulk.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,20 @@ It walks our local directory and produces a CSV manifest that maps each file to
the correct parent folder in Synapse (creating folders as needed). The output
is ready to hand directly to `sync_to_synapse`.

!!! warning "Folders are created in Synapse during manifest generation"
While walking the directory tree, `generate_sync_manifest` **creates the
corresponding folder structure on the project/folder it is called on**,
including any empty subfolders . Only the files themselves are deferred
to the later `sync_to_synapse` call. This is why the call still needs a
target Synapse Project/Folder even when you only want to produce a manifest
— each row in the manifest references the Synapse ID of its (already-created)
parent folder.

To upload a subset of files, generate the manifest first, delete the rows
for files you do not want to upload from `PATH_TO_MANIFEST_FILE`, then run
`sync_to_synapse`. The empty folders created during manifest generation
will remain in Synapse.

```python
--8<-- "docs/tutorials/python/tutorial_scripts/upload_data_in_bulk.py:generate_manifest"
```
Expand Down
28 changes: 23 additions & 5 deletions synapseclient/__main__.py
Original file line number Diff line number Diff line change
Expand Up @@ -1006,27 +1006,45 @@ def build_parser():
parser_get.set_defaults(func=get)

parser_manifest = subparsers.add_parser(
"manifest", help="Generate manifest for uploading directory tree to Synapse."
"manifest",
help="Generate manifest for uploading directory tree to Synapse.",
description=(
"Generate a TSV manifest describing the files in a local directory tree "
"for later upload with `synapse sync`. "
"IMPORTANT: this command will also create the corresponding folder structure "
"in Synapse while walking the directory under --parent-id "
"(including any empty folders); only the files are deferred to "
"the subsequent `synapse sync` call. "
"To upload a subset of files, run `synapse manifest` first, delete the rows "
"for files you do not want to upload from the generated manifest, then run "
"`synapse sync <manifest>`."
),
)
parser_manifest.add_argument(
"path",
metavar="PATH",
type=str,
help="A path to a file or folder whose manifest will be generated.",
help="Local path to the directory whose contents will be described in the "
"manifest. The directory tree is walked recursively.",
)
parser_manifest.add_argument(
"--parent-id",
metavar="syn123",
type=str,
required=True,
dest="parentid",
help="Synapse ID of project or folder where to upload data.",
help="Synapse ID of the project or folder that will be the parent of the "
"uploaded tree. Required even when only generating a manifest, because the "
"matching folder structure (including empty folders) is created in Synapse "
"under this parent as the manifest is generated, so that each row in the "
"manifest can reference the Synapse ID of its parent folder.",
)
parser_manifest.add_argument(
"--manifest-file",
metavar="OUTPUT",
help="A TSV output file path where the generated manifest is stored. "
"(default: stdout)",
help="Path where the generated TSV manifest will be written (for example "
"`./manifest.tsv`). If omitted, the manifest is printed to stdout; pass an "
"explicit path if you intend to edit it before running `synapse sync`.",
)
parser_manifest.set_defaults(func=manifest)

Expand Down
Loading