Describe the issue
databricks fs cp fails with "Bad Request" when copying a file to a UC managed volume that exists but is empty. The CLI performs a directory existence check via the Files API before writing, and an empty UC volume returns HTTP 404 for that check — causing the CLI to abort instead of writing the file.
Steps to reproduce the behavior
- Create a UC managed volume: databricks volumes create mycatalog myschema myvol MANAGED
- Upload a file to it: databricks fs cp myfile.xml dbfs:/Volumes/mycatalog/myschema/myvol/ (succeeds)
- Delete all files from the volume (e.g. via the Databricks UI)
- Re-run the same databricks fs cp command
Expected Behavior
The file is uploaded successfully. The volume object still exists; only its contents were deleted. A write operation to an existing volume should not depend on whether the volume currently has any files in it.
Actual Behavior
The command fails with Error: Bad Request. The CLI calls HEAD /api/2.0/fs/directories/Volumes/mycatalog/myschema/myvol before writing. An empty UC volume returns 404 for this endpoint. The CLI interprets this as "no such directory" and aborts without writing the file.
Debug trace:
HEAD /api/2.0/fs/directories/Volumes/mycatalog/myschema/myvol → 404 Not Found
HEAD /api/2.0/fs/files/Volumes/mycatalog/myschema/myvol → 404 Not Found
HEAD /api/2.0/fs/directories/Volumes/mycatalog/myschema → 400 Bad Request
Error: Bad Request
The volume object exists throughout — databricks volumes read mycatalog.myschema.myvol returns full metadata. The --overwrite flag does not help; the same error occurs.
Workaround: Use the Files REST API directly — PUT /api/2.0/fs/files/Volumes/mycatalog/myschema/myvol/myfile.xml — which writes unconditionally without a prior directory check.
OS and CLI version
- CLI: Databricks CLI v1.3.0
- OS: Linux 5.10.16.3 (WSL2 on Windows)
Is this a regression?
Unknown — this was first observed on v1.3.0. Not tested on earlier versions.
Debug Logs
GET /api/2.0/fs/directories/Volumes/mycatalog/myschema/myvol
< HTTP/2 404
HEAD /api/2.0/fs/directories/Volumes/mycatalog/myschema/myvol
< HTTP/2 404
HEAD /api/2.0/fs/files/Volumes/mycatalog/myschema/myvol
< HTTP/2 404
HEAD /api/2.0/fs/directories/Volumes/mycatalog/myschema
< HTTP/2 400
Error: Bad Request
Describe the issue
databricks fs cp fails with "Bad Request" when copying a file to a UC managed volume that exists but is empty. The CLI performs a directory existence check via the Files API before writing, and an empty UC volume returns HTTP 404 for that check — causing the CLI to abort instead of writing the file.
Steps to reproduce the behavior
Expected Behavior
The file is uploaded successfully. The volume object still exists; only its contents were deleted. A write operation to an existing volume should not depend on whether the volume currently has any files in it.
Actual Behavior
The command fails with Error: Bad Request. The CLI calls HEAD /api/2.0/fs/directories/Volumes/mycatalog/myschema/myvol before writing. An empty UC volume returns 404 for this endpoint. The CLI interprets this as "no such directory" and aborts without writing the file.
Debug trace:
HEAD /api/2.0/fs/directories/Volumes/mycatalog/myschema/myvol → 404 Not Found
HEAD /api/2.0/fs/files/Volumes/mycatalog/myschema/myvol → 404 Not Found
HEAD /api/2.0/fs/directories/Volumes/mycatalog/myschema → 400 Bad Request
Error: Bad Request
The volume object exists throughout — databricks volumes read mycatalog.myschema.myvol returns full metadata. The --overwrite flag does not help; the same error occurs.
Workaround: Use the Files REST API directly — PUT /api/2.0/fs/files/Volumes/mycatalog/myschema/myvol/myfile.xml — which writes unconditionally without a prior directory check.
OS and CLI version
Is this a regression?
Unknown — this was first observed on v1.3.0. Not tested on earlier versions.
Debug Logs
GET /api/2.0/fs/directories/Volumes/mycatalog/myschema/myvol
< HTTP/2 404
HEAD /api/2.0/fs/directories/Volumes/mycatalog/myschema/myvol
< HTTP/2 404
HEAD /api/2.0/fs/files/Volumes/mycatalog/myschema/myvol
< HTTP/2 404
HEAD /api/2.0/fs/directories/Volumes/mycatalog/myschema
< HTTP/2 400
Error: Bad Request