Repository navigation
Conversation
* feat(database): integrate Drizzle ORM with PostgreSQL 18 - Added PostgreSQL Helm chart and related Kubernetes configurations for local development and E2E testing. - Introduced new NPM scripts for database generation, migration, and management using Drizzle ORM. - Created database connection and migration logic in `drizzle.config.ts`, `db.ts`, and `migrate.ts`. - Defined user storage connections schema and initial migration files. - Updated `DATABASE.md` with setup instructions and usage guidelines for the new database system. * remove connection testing section from DATABASE.md * move DATABASE.md into docs folder
* feat: Add VSCode file icons and refactor FileRow component * #142 use materials and lsicon icons for CSV, ZIP and YAML --------- Co-authored-by: F3l1x1vo <fli@ordix.de>
Collaborator
|
@copilot resolve the merge conflicts in this pull request. Pull the changes from v0 (feat/s3-file-browser) into this v1 branch. |
…s3-file-browser-v1 # Conflicts: # package-lock.json # src/lib/server/storage/s3-provider.ts
Done. Merged Two conflicts were resolved:
|
Leftovers from the key-validation changes: `hasControlCharacter` was split out only so the key schema could use it, and the key schema's doc comment still described existing keys only. Inline the helper again and describe what the schema actually validates. The download manifest routes checked their keys by hand; they now use body schemas built on `StorageObjectKeySchema` with the same limits and error message. A `null` body now gets 400 instead of a 500 TypeError. Tests pin that typed names still reject control characters.
Typed create names accepted empty segments, so `/a` created the marker `prefix/` + `/`, `a//b` produced `a//` markers, and a file named `a/` silently became a folder marker. File or folder is chosen in the menu and the folder slash is added by the app, so names that start or end with "/" or contain "//" are now rejected with an inline error. The server still accepts any legal key. The download manifests POST also validates its `prefix` query parameter with `StoragePrefixSchema` instead of storing it unchecked.
…er API Copy, cut/paste, drag-move and rename were orchestrated by the client as several non-atomic requests. The server now plans and runs transfers itself: - POST /api/storage/transfer/plan previews destinations, conflicts, suggested names and refusals for the conflict dialog. - POST /api/storage/transfer runs one copy, move or rename for up to 1000 selected items and streams NDJSON per item (item-start, progress, item-done, complete). Statuses: done, partial, failed, refused, cancelled. - Folder conflicts merge (destination-only files are kept); file conflicts replace by overwriting, never by deleting first. Keep both and rename pick a free name at execution time; a rename never overwrites. - A move deletes exactly the keys it copied, only after every copy of that item succeeded. - Destinations that overlap a source are refused, including a folder into its own subtree or into an ancestor that contains it. Destination prefixes must be folders, and child keys over 1024 bytes are refused before anything is copied. - Copies run server-side with CopyObject, also across buckets of one connection; no data streams through the app. - Cancel aborts running S3 calls and keeps the originals of unfinished items. A client disconnect does not cancel; the client reads the item results back from the job, which a heartbeat keeps alive. - Objects per item are copied through a pool sized by STACKABLE_COCKPIT_STORAGE_TRANSFER_CONCURRENCY (default 8).
Bohreromir
force-pushed
the
feat/s3-file-browser-v1
branch
from
October 14, 2026 13:55
dfacc87 to
662417e
Compare
cockpit needs PostgreSQL for every install. The chart now takes database.host/port/name/user and a required password Secret reference, and a required storage.encryptionKey Secret reference when the storage browser is enabled. Database TLS is no longer forced in production without an opt-out. DATABASE_TLS_MODE selects verify-full (production default), require or disable, and DATABASE_TLS_CA_CERT points at a private CA.
The archive preview cache had no size cap, kept downloading after the client disconnected, skipped the size check when HEAD failed, downloaded the same archive once per concurrent request and wrote nested archives to disk outside any accounting. - Cap the cache at STACKABLE_COCKPIT_ARCHIVE_CACHE_MAX_MB (default 512 MB, at least twice the preview limit). Space is reserved before a download and idle archives are evicted LRU; readers hold leases, and a full cache answers 503. - Enforce the preview limit while streaming, not only via HEAD. - Share one download between concurrent requests and abort it once every waiting request has gone away. - Count nested archives in the cache. - Keep the cache in a per-process directory and remove directories of earlier processes on first use. - Add archive cache metrics. - Helm: set a sizeLimit on the /tmp emptyDir (tmp.sizeLimit, 1Gi) and expose config.archiveCacheMaxMb.
Calculating a directory size threw on large folders (Math.max spread over every key), recomputed the total on every batch, kept every key in memory, sent the whole tree in one message and kept listing after the client went away. - Aggregate each batch into running totals and a trie instead of keeping the keys; max depth comes from a loop. - Stop listing after 10 000 objects and mark the result as truncated; the UI shows a warning that the figures are partial. - Prune on the server: the treemap keeps the 30 largest files and groups the rest per folder into "N more" nodes, each depth list keeps the 1000 largest entries and reports what was left out. - Abort the S3 listing when the client disconnects, and cancel the request when the details modal closes mid-calculation. - Keep a file and a folder of the same name apart in the treemap. - Add the storage_directory_size_calculations_total metric.
The Parquet metadata cache and the archive disk cache were keyed by object key only, so after an overwrite the preview kept showing the old content until the entry expired. - Identify an object's version by its ETag, falling back to size and Last-Modified; an epoch Last-Modified counts as unknown. - Parquet: reuse cached metadata only while the version matches, and take the metadata from the preview route instead of another HEAD. - CSV: use the same version check for its line-offset cache. - Archives: HEAD on every request and key cache entries by version. Caching a new version removes the old ones. Without a version, or when the GET returns a different ETag than the HEAD, the download is used for that request only and removed afterwards, nested archives extracted from it included. - Record the real reason (single_use, stale, expired) in the archive cache eviction metric.
- Helm chart: expose infinite scroll, cut/copy, paste, rename, move, restore tabs, max editable file size, archive preview max size, Parquet disallowed compression types and download-history retention under features.storage. - Server re-exports the PUBLIC_ flags it enforces from the client module, so each is parsed once (infinite scroll was parsed differently on each side). ESLint allows that one client file. - Remove the auto-connect flag and its timeout: neither has been read since auto-connect moved to the session's active connection. - .env.example: list all storage settings, BODY_SIZE_LIMIT, the Trino public URL and query TTL; fix wrong comments; separate flags from limits.
…sible - Show invalid search filter values as visible text tied to the input via aria-describedby instead of colour and tooltip only. Drop role="alert" from the filter and max-depth errors so they don't interrupt typing. - DateTimePicker: keep the roving tab stop in the visible month after changing month, so Tab still reaches the day grid. - DateTimePicker: only return focus to the trigger on close when focus was inside the calendar or got lost, not when another control was clicked. - Breadcrumb count badges: hide the bare number from screen readers and add a full "3 folders" / "1 file" text alternative.
Nothing in the build or the app calls the zip binary: ZIP archives are written by zip-stream.ts and read by adm-zip, both pure JS. The final image is built from base, so the package never shipped either.
…ails checkObjectExists treated a failed HEAD (403, 5xx, network) as "no conflict", so the upload modal went ahead with "replace" and could silently overwrite an existing object. Now any failed check uploads nothing, shows an error and returns focus to Upload for a retry. Also drop the dead `return res.ok` in both checkObjectExists copies: storageFetch already throws on any non-2xx response.
…ed writes getConnectionFromHeader now parses the decrypted payload with StoredStorageConnectionSchema and validates additionalBuckets instead of casting. updated_at is only refreshed when it is older than five minutes rather than on every storage API request.
…validation - Cancelled searches show "Cancelled" instead of "Done" and no longer count as completed. - A max folder depth of 0 or non-numeric input is rejected instead of silently meaning unlimited. - Search result and download history dates follow the app locale. - Exclude patterns match case-insensitively. - Search history, download manifest and redownload endpoints return 400 on malformed JSON; search history validates its body with a zod schema.
…T errors Bucket info getters returned defaults on any error, so access denied or unsupported calls showed as "Disabled", owner "Unknown", no rules or no tags. Only the "not configured" codes now mean empty; other failures return null and the details modals show "Could not load" for just that section. The zero-length PutObject path skipped S3 error mapping, so a 403 from S3 reached the provider proxy and came back as 502.
Objects over 5 GiB are copied with UploadPartCopy. Without a precondition, overwriting the source mid-copy produced a destination mixing old and new parts while the copy reported success; a move then deleted the source. Every part now sends CopySourceIfMatch with the ETag from the initial HEAD, so a changed source fails the copy with 412, the multipart upload is aborted and a move keeps its original.
…HEAD The transfer concurrency only applied within one item, so pasting many single files ran fully serially. Items now run through the pool, and one shared limiter caps in-flight copies across the whole request. copyObject takes the source size when the caller knows it and skips the HEAD for objects up to 5 GiB; multipart copies still read the source for the ETag pin. A vanished source is reported by its own key. The client tracks progress per running item, since item events now interleave and arrive in completion order.
"Check new name" always reported the name as free: the dialog no longer got a checker and fell back to "available". The server then refused the item with a conflict after the dialog had closed. The plan endpoint now takes optional new names per source key and plans those items as renamed, under the same rules as the transfer itself, so folders without a marker object count as taken. The dialog checks a name with a one-item plan, and also rejects names that items without a conflict in the same paste land on. A failed check shows an error and lets the user retry. A result that arrives after the user changed the choice or the name is ignored. The checker is now required.
Workers in a browser project shared one storageState session, so parallel storage tests disconnected each other and deleted each other's saved connections. A worker-scoped fixture now logs each worker in as its own mock OIDC user (Playwright's "one account per parallel worker" pattern), replacing the setup-* projects. With that, run fully parallel with 4 workers locally and 2 on CI, and replace fixed sleeps in the storage specs with condition waits.
listAllKeysProgressively claimed to prefetch the next page while a batch was processed, but it only sent the next request after the synchronous callback returned, so nothing overlapped. Every caller's callback is synchronous and cheap, so real prefetching would gain nothing either. Replace the fake pipeline with a private listPages generator that both listAllKeysProgressively and listAllKeys iterate, and pin the empty-page, callback-error and abort-signal behaviour with tests.
…errors deleteObjects listed every key under each directory into memory before deleting anything, which does not scale to directories with millions of objects. The listing also ran outside withS3Errors, so a 403 or 404 while listing surfaced as an unmapped 500. Delete each full chunk of 1000 keys while the listing continues, so at most about two chunks of keys are held at once, and map every S3 error from the operation. A listing failure now leaves the chunks deleted before it in place; deletes were never atomic across chunks anyway.
A delete that fails part way, for example on a later listing page, may already have removed some of the selection. The error path only showed a toast, so deleted objects stayed visible until a manual reload.
deleteObjects no longer uses it, so it had no production callers left, and it loads every key under a prefix into memory. Callers that need a recursive listing use listAllKeysProgressively. Its remaining test cases now cover the progressive listing.
S3 connections with TLS verification could only trust the CAs bundled with Node.js, so on-prem stores with an internal CA had to disable verification. STACKABLE_COCKPIT_STORAGE_TLS_CA_CERT, set via the Helm values storage.tls.caCert or storage.tls.secretClass, now names a CA certificate that replaces the bundled CAs for every verified S3 connection, like the existing Trino and database CA settings.
…characters The rename modal reused the conflict dialog's message key. Rename both invalid-name keys into the storage_name_* family and mention control characters, which the name policy already rejects.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #129