Support bucketless Benchling webhook deployments

[drernie]

Summary

• make the package bucket optional across setup/config/secrets/deploy paths
• skip default package creation for entry and canvas events when no bucket is configured
• search all Quilt package-view buckets for linked packages in bucketless mode and preserve bucket context for browsing/metadata
• bump package version to 0.19.0 and document the release notes

Tests

• npm test
• npm run version:verify
• npm run test:local built and launched the Docker dev container, but webhook smoke checks failed because the local dev profile is degraded: PACKAGING_REQUEST_QUEUE_URL is not configured

Greptile Summary

This PR makes the Quilt package bucket optional across setup, secret creation, deployment, and runtime, enabling "bucketless" deployments that skip default package creation for entry/canvas events and instead search all available Quilt package-view buckets when resolving linked packages.

• Runtime guards (app.py, entry_packager.py, sqs_consumer.py): every handler that would auto-create a package now returns a 202 SKIPPED response when s3_bucket_name is empty; SQS consumer passes the event's actual bucket through to canvas refresh rather than filtering on a configured bucket.
• Multi-bucket linked-package discovery (package_query.py, canvas.py, canvas_blocks.py, pagination.py): in bucketless mode the Athena query enumerates all *_packages-view tables and merges results; browse/metadata button IDs now embed an optional -bucket-{name} segment so the correct PackageFileFetcher can be reconstructed on subsequent canvas interactions.
• Configuration/deployment (lib/, bin/): bucket made optional in PackageConfig, wizard parameter collection, profile builders, secret sync, and IAM policy synthesis; configuration-validator.ts updated but validateConfig in lib/utils/config.ts was not.

Confidence Score: 3/5

The bucket-configured path is unchanged and safe; the bucketless path has two defects that would silently produce wrong behavior for users with non-standard bucket names or who reach the legacy validateConfig function.

The encode/decode scheme for bucket names replaces '--' with '/' on decode, corrupting any S3 bucket name with consecutive hyphens. Separately, validateConfig in lib/utils/config.ts still rejects a missing quiltUserBucket even though its description was updated to call the field optional.

docker/src/pagination.py (encode/decode collision), lib/utils/config.ts (requiredUserFields still includes quiltUserBucket)

Important Files Changed

Filename	Overview
docker/src/pagination.py	Added encode/decode functions for bucket names and a new 5-tuple parse function; the decode_bucket_name function has a collision bug for bucket names containing '--', and the docstring for the new function was not updated.
lib/utils/config.ts	quiltUserBucket description updated to say 'Optional' but the field remains in the requiredUserFields array, meaning validateConfig() still errors when the bucket is absent.
docker/src/package_query.py	Extracted single-bucket query to _find_unique_packages_in_bucket; added _list_package_view_buckets for bucketless multi-bucket search; LIKE pattern uses unescaped _ wildcard (minor, corrected by Python endswith filter).
docker/src/canvas.py	Gracefully handles bucketless mode in markdown rendering and navigation buttons; adds _file_fetcher_for_bucket to create per-bucket fetchers when browsing linked packages across multiple buckets.
docker/src/app.py	Adds _bucketless_response helper and guards all packaging-trigger handlers to skip default package creation when s3_bucket_name is empty.
docker/src/canvas_blocks.py	Extended browser/metadata navigation button constructors to accept optional bucket_name; relies on the encode/decode scheme that has the -- collision risk.
docker/src/secrets_manager.py	user_bucket moved to an optional dataclass field with default None; removed from required-params validation lists.
docker/src/sqs_consumer.py	Bucket filter is now conditional on s3_bucket_name being set; passes parsed.bucket through to refresh_canvas_for_package_event in bucketless mode.
lib/fargate-service.ts	packageBucket is now optional; S3 IAM policy block wrapped in if(props.packageBucket) guard to avoid empty-ARN policy statements.
lib/configuration-validator.ts	quiltUserBucket removed from the required-fields list, correctly making it optional for bucketless deployments.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Webhook Event] --> B{s3_bucket_name set?}
    B -- Yes --> C[Publish SQS packaging request]
    B -- No --> D[Return 202 SKIPPED bucketless response]
    C --> E[SqsConsumer receives Quilt event]
    E --> F{s3_bucket_name set AND bucket mismatch?}
    F -- Yes --> G[Skip / delete message]
    F -- No --> H[refresh_canvas_for_package_event with parsed.bucket]
    I[Canvas button click] --> J{Action type?}
    J -- Browse/Update/Metadata default --> K{s3_bucket_name set?}
    K -- No --> D
    K -- Yes --> L[Use configured bucket]
    J -- Browse Linked --> M[parse button ID, extract bucket_name]
    M --> N[_file_fetcher_for_bucket bucket_name]
    N --> O{bucket_name matches default fetcher?}
    O -- Yes --> P[Reuse _package_file_fetcher]
    O -- No --> Q[Create new PackageFileFetcher for that bucket]
    R[PackageQuery.find_unique_packages] --> S{self.bucket set?}
    S -- Yes --> T[_find_unique_packages_in_bucket self.bucket]
    S -- No --> U[_list_package_view_buckets Athena information_schema]
    U --> V[For each bucket: _find_unique_packages_in_bucket]
    V --> W[Merge results sorted by bucket+name]

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A[Webhook Event] --> B{s3_bucket_name set?}
    B -- Yes --> C[Publish SQS packaging request]
    B -- No --> D[Return 202 SKIPPED bucketless response]
    C --> E[SqsConsumer receives Quilt event]
    E --> F{s3_bucket_name set AND bucket mismatch?}
    F -- Yes --> G[Skip / delete message]
    F -- No --> H[refresh_canvas_for_package_event with parsed.bucket]
    I[Canvas button click] --> J{Action type?}
    J -- Browse/Update/Metadata default --> K{s3_bucket_name set?}
    K -- No --> D
    K -- Yes --> L[Use configured bucket]
    J -- Browse Linked --> M[parse button ID, extract bucket_name]
    M --> N[_file_fetcher_for_bucket bucket_name]
    N --> O{bucket_name matches default fetcher?}
    O -- Yes --> P[Reuse _package_file_fetcher]
    O -- No --> Q[Create new PackageFileFetcher for that bucket]
    R[PackageQuery.find_unique_packages] --> S{self.bucket set?}
    S -- Yes --> T[_find_unique_packages_in_bucket self.bucket]
    S -- No --> U[_list_package_view_buckets Athena information_schema]
    U --> V[For each bucket: _find_unique_packages_in_bucket]
    V --> W[Merge results sorted by bucket+name]

Loading

Comments Outside Diff (2)

1. lib/utils/config.ts, line 203-223 (link)

   validateConfig still treats quiltUserBucket as required despite the "Optional" description

   quiltUserBucket remains in the requiredUserFields array, so the loop on line 215 pushes an error whenever the bucket is absent. The newly changed helpText reads "Optional. When omitted, bucketless mode…" but the code still produces errors.push(...) for that field, which sets valid: false. Any caller that uses the return value to gate a bucketless deployment would be incorrectly blocked. lib/configuration-validator.ts was correctly updated to remove the field from its required list; this utility function was not.

2. docker/src/pagination.py, line 61-84 (link)

   Docstring for parse_browse_linked_button_id_with_bucket describes the old 4-tuple return

   The Returns: section still says "Tuple of (entry_id, package_name, page_number, page_size)" and the examples reference the old parse_browse_linked_button_id function instead of the new one. The actual return type is a 5-tuple (entry_id, package_name, bucket_name, page_number, page_size). Doctests or documentation tooling that exercises these examples would fail.

_{Reviews (1): Last reviewed commit: "docs: update changelog for 0.19.0" | Re-trigger Greptile}

Greptile also left 2 inline comments on this PR.

[Copilot]

Pull request overview

This PR adds “bucketless” deployment support for the Benchling webhook by making the package bucket optional throughout setup, validation, secret sync, deployment/CDK wiring, and runtime behavior. In bucketless mode, default package creation is skipped for entry/canvas events and linked-package discovery broadens to search across available Quilt buckets (with optional Iceberg-backed acceleration via a new IcebergDatabase/QUILT_ICEBERG_DATABASE path).

Changes:

• Make packages.bucket optional across TypeScript config/schema, wizard/profile building, deploy/validate/sync-secret commands, and the Python runtime secret/config model.
• Update Python runtime handlers and Canvas UI flows to behave safely in bucketless mode (skip default package creation; preserve per-linked-package bucket context for browsing/metadata).
• Add optional Iceberg database propagation and tests, plus a version bump to 0.19.0 with release notes/spec documentation.

Reviewed changes

Copilot reviewed 56 out of 64 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
test/utils-stack-inference.test.ts	Adds coverage for extracting Iceberg DB output without overwriting Quilt DB.
test/unit/service-resolver.test.ts	Ensures optional Iceberg DB output is surfaced in resolved services.
test/unit/config-types.test.ts	Validates icebergDatabase typing and JSON schema acceptance.
test/sync-secrets.test.ts	Adds dry-run redaction test to prevent printing plaintext client secrets.
test/integration/xdg-launch-pure-functions.test.ts	Extends inferred resource extraction to include Iceberg DB.
test/config-transform.test.ts	Adds bucketless profile validation and threads icebergDatabase through transforms.
test/benchling-webhook-stack.test.ts	Verifies Iceberg parameter/env propagation and Glue policy expectations in synthesized templates.
spec/a10-bucketless/README.md	Adds spec workspace stub for bucketless workstream.
spec/a10-bucketless/06-iceberg-parameter-audit.md	Documents Iceberg parameter propagation gaps/requirements (spec).
spec/a10-bucketless/05-iceberg-search.md	Proposes Iceberg-backed bucketless linked-package search (spec).
spec/a10-bucketless/04-bucketless-next-steps-and-blockers.md	Summarizes bucketless status and known blockers (spec).
spec/a10-bucketless/03-bucketless-checklist.md	Adds implementation/deployment checklist for bucketless mode (spec).
spec/a10-bucketless/02-bucketless-audit.md	Audits impacted code paths for bucketless mode (spec).
spec/a10-bucketless/01-bucketless-req.md	Defines bucketless user requirements and acceptance criteria (spec).
scripts/check-iceberg-search.sh	Adds an end-to-end probe script for Iceberg-backed bucketless search.
package.json	Bumps package version to 0.19.0.
package-lock.json	Aligns lockfile version fields with 0.19.0.
lib/wizard/types.ts	Makes wizard-collected packages.bucket optional; adds optional icebergDatabase to stack query result.
lib/wizard/profile-config-builder.ts	Persists optional bucket and discovered Iceberg DB into profiles.
lib/wizard/phase7-standalone-mode.ts	Avoids writing packages.bucket into bucketless profiles.
lib/wizard/phase6-integrated-mode.ts	Avoids writing packages.bucket into bucketless profiles.
lib/wizard/phase4-validation.ts	Skips S3 bucket validation when bucketless; updates output wording.
lib/wizard/phase3-parameter-collection.ts	Supports bucketless parameter collection and interactive “clear bucket” sentinel.
lib/wizard/phase2-stack-query.ts	Displays and returns discovered Iceberg DB.
lib/utils/stack-inference.ts	Adds Iceberg resource/output extraction into inferred env vars.
lib/utils/service-resolver.ts	Reads optional Iceberg DB outputs into QuiltServices.
lib/utils/config.ts	Removes bucket from required legacy user fields while retaining bucket-format warning when present.
lib/utils/config-transform.ts	Copies optional icebergDatabase from profile to stack config.
lib/types/stack-config.ts	Adds quilt.icebergDatabase to typed stack config.
lib/types/config.ts	Makes packages.bucket optional; adds quilt.icebergDatabase; updates profile schema requirements.
lib/fargate-service.ts	Makes package bucket optional; adds Glue + Iceberg-related permissions and env var propagation.
lib/configuration-validator.ts	Removes quiltUserBucket from required legacy validator fields.
lib/benchling-webhook-stack.ts	Adds optional IcebergDatabase parameter and passes through to service construct.
docker/uv.lock	Bumps Python package version to 0.19.0.
docker/tests/test_package_query.py	Adds bucketless multi-bucket and Iceberg-path unit tests for package lookup.
docker/tests/test_config_validation.py	Makes user_bucket optional in secrets and tests bucketless parsing.
docker/tests/test_canvas_updating_blocks.py	Adds bucketless Canvas rendering/update-state tests.
docker/tests/test_browse_linked_packages.py	Updates linked browse button text and adds bucketless nav tests.
docker/tests/test_app.py	Adds bucketless event/canvas handler behavior tests including refresh path.
docker/tests/conftest.py	Adds QUILT_ICEBERG_DATABASE to isolated env fixture.
docker/src/sqs_consumer.py	Makes bucket filtering conditional and forwards event bucket to refresh.
docker/src/secrets_manager.py	Makes user_bucket optional in secret model and validation.
docker/src/pagination.py	Adds button-ID bucket encoding/parsing support for linked-package browsing.
docker/src/package_query.py	Adds bucketless multi-bucket lookup and optional Iceberg single-query search path.
docker/src/package_event.py	Allows refresh to use the event bucket in bucketless mode.
docker/src/entry_packager.py	Skips entry packaging workflow entirely when bucketless.
docker/src/config.py	Adds quilt_iceberg_database env config; treats missing user_bucket as bucketless.
docker/src/config_schema.py	Makes user_bucket optional in the Pydantic secret schema.
docker/src/canvas.py	Updates Canvas behavior for bucketless mode and per-linked-package bucket browsing/metadata.
docker/src/canvas_formatting.py	Adjusts “package not found” formatting to support non-creatable (bucketless) mode.
docker/src/canvas_blocks.py	Adds bucketless navigation controls and embeds optional bucket info into linked button IDs.
docker/src/app.py	Adds bucketless guards/response helpers and a refresh-canvas handler.
docker/scripts/quilt_search_probe.py	Adds an external probe script for Quilt search behavior across buckets.
docker/pyproject.toml	Bumps Python package version to 0.19.0.
docker/docker-compose.yml	Propagates PACKAGING_REQUEST_QUEUE_URL into compose services.
docker/app-manifest.yaml	Bumps app manifest version to 0.19.0.
CHANGELOG.md	Adds 0.19.0 release notes for bucketless support.
bin/xdg-launch.ts	Loads local .env for convenience and propagates QUILT_ICEBERG_DATABASE.
bin/commands/validate.ts	Prints bucketless mode clearly when bucket is absent.
bin/commands/sync-secrets.ts	Omits user_bucket when bucketless and masks client secret in dry-run output.
bin/commands/infer-quilt-config.ts	Refines stack output parsing and extracts Iceberg DB explicitly.
bin/commands/deploy.ts	Supports bucketless deploy parameters and passes optional Iceberg DB through.
bin/commands/create-secret.ts	Makes BENCHLING_USER_BUCKET optional and masks client secret in dry-run.
AGENTS.md	Clarifies dev vs release tagging scripts and workflows.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.