Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
f09b6f6
testserver: add storage for postgres synced tables
pietern May 18, 2026
9688b99
testserver: add postgres synced table CRUD
pietern May 18, 2026
f0036ce
testserver: route postgres synced table endpoints
pietern May 18, 2026
721a598
bundle: add PostgresSyncedTable config resource type
pietern May 18, 2026
3c1f4c8
dresources: add TrimSyncedTablesPrefix helper
pietern May 18, 2026
9037247
dresources: implement postgres synced table handler
pietern May 18, 2026
53029c9
bundle: register postgres_synced_tables resource
pietern May 18, 2026
f39bd3c
dresources: declare recreate_on_changes for postgres synced tables
pietern May 18, 2026
5ccf8d1
dresources: add postgres synced table smoke test
pietern May 18, 2026
bb87545
schema: document postgres_synced_tables
pietern May 18, 2026
5d6d658
acc: postgres synced table basic deploy
pietern May 18, 2026
de011f9
acc: postgres synced table recreate on spec change
pietern May 18, 2026
2adce5e
dresources: register postgres_synced_tables in knownMissingInRemoteType
pietern May 18, 2026
a3b1372
statemgmt: cover postgres_synced_tables in state-load fixtures
pietern May 18, 2026
6bbe5ac
terraform: mark postgres_synced_tables as direct-only in lifecycle test
pietern May 18, 2026
f225351
resourcemutator: declare postgres_synced_tables in permissions/run_as…
pietern May 18, 2026
52913dc
schema: write postgres_synced_tables annotation
pietern May 18, 2026
e388091
statemgmt: use prefixed API name as postgres_synced_tables state ID
pietern May 18, 2026
5732819
changelog: postgres_synced_tables bundle resource
pietern May 18, 2026
f26197d
dresources: fix forever-recreate for postgres_synced_tables
pietern May 18, 2026
ca14073
acc: cover postgres_synced_tables with no_drift invariant
pietern May 18, 2026
2beb3cb
Add postgres_catalogs bundle resource
pietern May 18, 2026
9744962
Merge branch 'postgres-catalog' into postgres-synced-table
pietern May 18, 2026
c4e5908
bundle: add terraform support for postgres_synced_tables
pietern May 18, 2026
9150e27
acc: make postgres_synced_tables cloud-deployable end to end
pietern May 19, 2026
376a4a9
acc: manage pipeline storage schema as a bundle resource
pietern May 19, 2026
7045564
acc: per-test source table for postgres_synced_tables
pietern May 19, 2026
495a076
acc: normalize synced-table GET output for cloud + local
pietern May 19, 2026
5d73765
Merge branch 'postgres-catalog' into postgres-synced-table
pietern May 19, 2026
73eabf1
changelog: align postgres_synced_tables wording with docs
pietern May 19, 2026
36ae5e8
validation: regenerate enum + required fields for postgres_synced_tables
pietern May 19, 2026
e8f59da
dresources: mirror postgres_catalogs pattern for synced_table_id
pietern May 19, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions NEXT_CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,5 +11,6 @@
### Bundles
* Make sure warnings asking for approval are understood by agents ([#5239](https://github.com/databricks/cli/pull/5239))
* Add `postgres_catalogs` resource to bind a Unity Catalog catalog to a Postgres database on a Lakebase Autoscaling branch ([#5265](https://github.com/databricks/cli/pull/5265)).
* Add `postgres_synced_tables` resource to sync a Unity Catalog Delta table into a Postgres table on a Lakebase Autoscaling branch ([#5268](https://github.com/databricks/cli/pull/5268)).

### Dependency updates
16 changes: 16 additions & 0 deletions acceptance/bundle/invariant/configs/postgres_synced_table.yml.tmpl
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
bundle:
name: test-bundle-$UNIQUE_NAME

resources:
postgres_synced_tables:
foo:
synced_table_id: lakebase_$UNIQUE_NAME.public.trips_synced
source_table_full_name: main.raw.trips
primary_key_columns: [id]
scheduling_policy: SNAPSHOT
postgres_database: appdb
branch: projects/test-pg-project-$UNIQUE_NAME/branches/production
create_database_objects_if_missing: true
new_pipeline_spec:
storage_catalog: main
storage_schema: pipelines
1 change: 1 addition & 0 deletions acceptance/bundle/invariant/continue_293/out.test.toml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions acceptance/bundle/invariant/migrate/out.test.toml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions acceptance/bundle/invariant/no_drift/out.test.toml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions acceptance/bundle/invariant/test.toml
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ EnvMatrix.INPUT_CONFIG = [
"postgres_catalog.yml.tmpl",
"postgres_endpoint.yml.tmpl",
"postgres_project.yml.tmpl",
"postgres_synced_table.yml.tmpl",
"registered_model.yml.tmpl",
"schema.yml.tmpl",
"schema_grant_ref.yml.tmpl",
Expand All @@ -69,6 +70,7 @@ no_postgres_project_on_cloud = ["CONFIG_Cloud=true", "INPUT_CONFIG=postgres_proj
no_postgres_branch_on_cloud = ["CONFIG_Cloud=true", "INPUT_CONFIG=postgres_branch.yml.tmpl"]
no_postgres_endpoint_on_cloud = ["CONFIG_Cloud=true", "INPUT_CONFIG=postgres_endpoint.yml.tmpl"]
no_postgres_catalog_on_cloud = ["CONFIG_Cloud=true", "INPUT_CONFIG=postgres_catalog.yml.tmpl"]
no_postgres_synced_table_on_cloud = ["CONFIG_Cloud=true", "INPUT_CONFIG=postgres_synced_table.yml.tmpl"]

# External locations require actual storage credentials with cloud IAM setup
# which are environment-specific, so we only test locally with the mock server
Expand Down
57 changes: 57 additions & 0 deletions acceptance/bundle/refschema/out.fields.txt
Original file line number Diff line number Diff line change
Expand Up @@ -2892,6 +2892,63 @@ resources.postgres_projects.*.permissions[*].group_name string ALL
resources.postgres_projects.*.permissions[*].level iam.PermissionLevel ALL
resources.postgres_projects.*.permissions[*].service_principal_name string ALL
resources.postgres_projects.*.permissions[*].user_name string ALL
resources.postgres_synced_tables.*.branch string INPUT STATE
resources.postgres_synced_tables.*.create_database_objects_if_missing bool INPUT STATE
resources.postgres_synced_tables.*.create_time *time.Time REMOTE
resources.postgres_synced_tables.*.existing_pipeline_id string INPUT STATE
resources.postgres_synced_tables.*.id string INPUT
resources.postgres_synced_tables.*.lifecycle resources.Lifecycle INPUT
resources.postgres_synced_tables.*.lifecycle.prevent_destroy bool INPUT
resources.postgres_synced_tables.*.modified_status string INPUT
resources.postgres_synced_tables.*.name string REMOTE
resources.postgres_synced_tables.*.new_pipeline_spec *postgres.NewPipelineSpec INPUT STATE
resources.postgres_synced_tables.*.new_pipeline_spec.budget_policy_id string INPUT STATE
resources.postgres_synced_tables.*.new_pipeline_spec.storage_catalog string INPUT STATE
resources.postgres_synced_tables.*.new_pipeline_spec.storage_schema string INPUT STATE
resources.postgres_synced_tables.*.postgres_database string INPUT STATE
resources.postgres_synced_tables.*.primary_key_columns []string INPUT STATE
resources.postgres_synced_tables.*.primary_key_columns[*] string INPUT STATE
resources.postgres_synced_tables.*.scheduling_policy postgres.SyncedTableSyncedTableSpecSyncedTableSchedulingPolicy INPUT STATE
resources.postgres_synced_tables.*.source_table_full_name string INPUT STATE
resources.postgres_synced_tables.*.spec *postgres.SyncedTableSyncedTableSpec REMOTE
resources.postgres_synced_tables.*.spec.branch string REMOTE
resources.postgres_synced_tables.*.spec.create_database_objects_if_missing bool REMOTE
resources.postgres_synced_tables.*.spec.existing_pipeline_id string REMOTE
resources.postgres_synced_tables.*.spec.new_pipeline_spec *postgres.NewPipelineSpec REMOTE
resources.postgres_synced_tables.*.spec.new_pipeline_spec.budget_policy_id string REMOTE
resources.postgres_synced_tables.*.spec.new_pipeline_spec.storage_catalog string REMOTE
resources.postgres_synced_tables.*.spec.new_pipeline_spec.storage_schema string REMOTE
resources.postgres_synced_tables.*.spec.postgres_database string REMOTE
resources.postgres_synced_tables.*.spec.primary_key_columns []string REMOTE
resources.postgres_synced_tables.*.spec.primary_key_columns[*] string REMOTE
resources.postgres_synced_tables.*.spec.scheduling_policy postgres.SyncedTableSyncedTableSpecSyncedTableSchedulingPolicy REMOTE
resources.postgres_synced_tables.*.spec.source_table_full_name string REMOTE
resources.postgres_synced_tables.*.spec.timeseries_key string REMOTE
resources.postgres_synced_tables.*.status *postgres.SyncedTableSyncedTableStatus REMOTE
resources.postgres_synced_tables.*.status.detailed_state postgres.SyncedTableState REMOTE
resources.postgres_synced_tables.*.status.last_processed_commit_version int64 REMOTE
resources.postgres_synced_tables.*.status.last_sync *postgres.SyncedTablePosition REMOTE
resources.postgres_synced_tables.*.status.last_sync.delta_table_sync_info *postgres.DeltaTableSyncInfo REMOTE
resources.postgres_synced_tables.*.status.last_sync.delta_table_sync_info.delta_commit_time *time.Time REMOTE
resources.postgres_synced_tables.*.status.last_sync.delta_table_sync_info.delta_commit_version int64 REMOTE
resources.postgres_synced_tables.*.status.last_sync.sync_end_time *time.Time REMOTE
resources.postgres_synced_tables.*.status.last_sync.sync_start_time *time.Time REMOTE
resources.postgres_synced_tables.*.status.last_sync_time *time.Time REMOTE
resources.postgres_synced_tables.*.status.message string REMOTE
resources.postgres_synced_tables.*.status.ongoing_sync_progress *postgres.SyncedTablePipelineProgress REMOTE
resources.postgres_synced_tables.*.status.ongoing_sync_progress.estimated_completion_time_seconds float64 REMOTE
resources.postgres_synced_tables.*.status.ongoing_sync_progress.latest_version_currently_processing int64 REMOTE
resources.postgres_synced_tables.*.status.ongoing_sync_progress.sync_progress_completion float64 REMOTE
resources.postgres_synced_tables.*.status.ongoing_sync_progress.synced_row_count int64 REMOTE
resources.postgres_synced_tables.*.status.ongoing_sync_progress.total_row_count int64 REMOTE
resources.postgres_synced_tables.*.status.pipeline_id string REMOTE
resources.postgres_synced_tables.*.status.project string REMOTE
resources.postgres_synced_tables.*.status.provisioning_phase postgres.ProvisioningPhase REMOTE
resources.postgres_synced_tables.*.status.unity_catalog_provisioning_state postgres.ProvisioningInfoState REMOTE
resources.postgres_synced_tables.*.synced_table_id string INPUT STATE
resources.postgres_synced_tables.*.timeseries_key string INPUT STATE
resources.postgres_synced_tables.*.uid string REMOTE
resources.postgres_synced_tables.*.url string INPUT
resources.quality_monitors.*.assets_dir string ALL
resources.quality_monitors.*.baseline_table_name string ALL
resources.quality_monitors.*.custom_metrics []catalog.MonitorMetric ALL
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
bundle:
name: deploy-postgres-synced-table-$UNIQUE_NAME

sync:
paths: []

resources:
schemas:
pipeline_storage:
name: pipeline_storage_$UNIQUE_NAME
catalog_name: main
comment: "Pipeline storage for the synced-table test"

postgres_projects:
my_project:
project_id: test-pg-proj-$UNIQUE_NAME
display_name: "Test Project for Synced Table"
pg_version: 17

postgres_catalogs:
my_catalog:
catalog_id: lakebase_test_$UNIQUE_NAME
branch: ${resources.postgres_projects.my_project.id}/branches/production
postgres_database: appdb
create_database_if_missing: true

postgres_synced_tables:
my_table:
synced_table_id: ${resources.postgres_catalogs.my_catalog.catalog_id}.public.trips_synced
source_table_full_name: main.source_$UNIQUE_NAME.trips_source
primary_key_columns: ["tpep_pickup_datetime"]
scheduling_policy: SNAPSHOT
postgres_database: appdb
branch: ${resources.postgres_projects.my_project.id}/branches/production
create_database_objects_if_missing: true
new_pipeline_spec:
storage_catalog: ${resources.schemas.pipeline_storage.catalog_name}
storage_schema: ${resources.schemas.pipeline_storage.name}
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
{
"method": "POST",
"path": "/api/2.0/postgres/projects",
"q": {
"project_id": "test-pg-proj-[UNIQUE_NAME]"
},
"body": {
"spec": {
"display_name": "Test Project for Synced Table",
"pg_version": 17
}
}
}
{
"method": "POST",
"path": "/api/2.0/postgres/catalogs",
"q": {
"catalog_id": "lakebase_test_[UNIQUE_NAME]"
},
"body": {
"spec": {
"branch": "projects/test-pg-proj-[UNIQUE_NAME]/branches/production",
"create_database_if_missing": true,
"postgres_database": "appdb"
}
}
}
{
"method": "POST",
"path": "/api/2.0/postgres/synced_tables",
"q": {
"synced_table_id": "lakebase_test_[UNIQUE_NAME].public.trips_synced"
},
"body": {
"spec": {
"branch": "projects/test-pg-proj-[UNIQUE_NAME]/branches/production",
"create_database_objects_if_missing": true,
"new_pipeline_spec": {
"storage_catalog": "main",
"storage_schema": "pipeline_storage_[UNIQUE_NAME]"
},
"postgres_database": "appdb",
"primary_key_columns": [
"tpep_pickup_datetime"
],
"scheduling_policy": "SNAPSHOT",
"source_table_full_name": "main.source_[UNIQUE_NAME].trips_source"
}
}
}
{
"method": "GET",
"path": "/api/2.0/postgres/synced_tables/lakebase_test_[UNIQUE_NAME].public.trips_synced"
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
{
"method": "POST",
"path": "/api/2.0/postgres/projects",
"q": {
"project_id": "test-pg-proj-[UNIQUE_NAME]"
},
"body": {
"spec": {
"display_name": "Test Project for Synced Table",
"pg_version": 17
}
}
}
{
"method": "POST",
"path": "/api/2.0/postgres/catalogs",
"q": {
"catalog_id": "lakebase_test_[UNIQUE_NAME]"
},
"body": {
"spec": {
"branch": "projects/test-pg-proj-[UNIQUE_NAME]/branches/production",
"create_database_if_missing": true,
"postgres_database": "appdb"
}
}
}
{
"method": "POST",
"path": "/api/2.0/postgres/synced_tables",
"q": {
"synced_table_id": "lakebase_test_[UNIQUE_NAME].public.trips_synced"
},
"body": {
"spec": {
"branch": "projects/test-pg-proj-[UNIQUE_NAME]/branches/production",
"create_database_objects_if_missing": true,
"new_pipeline_spec": {
"storage_catalog": "main",
"storage_schema": "pipeline_storage_[UNIQUE_NAME]"
},
"postgres_database": "appdb",
"primary_key_columns": [
"tpep_pickup_datetime"
],
"scheduling_policy": "SNAPSHOT",
"source_table_full_name": "main.source_[UNIQUE_NAME].trips_source"
}
}
}
{
"method": "GET",
"path": "/api/2.0/postgres/synced_tables/lakebase_test_[UNIQUE_NAME].public.trips_synced"
}

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
Creating temporary source table: main.source_[UNIQUE_NAME].trips_source
{
"full_name": "main.source_[UNIQUE_NAME]"
}

>>> [CLI] bundle validate
Name: deploy-postgres-synced-table-[UNIQUE_NAME]
Target: default
Workspace:
User: [USERNAME]
Path: /Workspace/Users/[USERNAME]/.bundle/deploy-postgres-synced-table-[UNIQUE_NAME]/default

Validation OK!

>>> [CLI] bundle summary
Name: deploy-postgres-synced-table-[UNIQUE_NAME]
Target: default
Workspace:
User: [USERNAME]
Path: /Workspace/Users/[USERNAME]/.bundle/deploy-postgres-synced-table-[UNIQUE_NAME]/default
Resources:
Postgres catalogs:
my_catalog:
Name: lakebase_test_[UNIQUE_NAME]
URL: [DATABRICKS_URL]/explore/data/lakebase_test_[UNIQUE_NAME]
Postgres projects:
my_project:
Name: Test Project for Synced Table
URL: (not deployed)
Postgres synced tables:
my_table:
Name: ${resources.postgres_catalogs.my_catalog.catalog_id}.public.trips_synced
URL: [DATABRICKS_URL]/explore/data/$%7Bresources.postgres_catalogs.my_catalog.catalog_id%7D.public.trips_synced
Schemas:
pipeline_storage:
Name: pipeline_storage_[UNIQUE_NAME]
URL: (not deployed)

>>> [CLI] bundle deploy
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/deploy-postgres-synced-table-[UNIQUE_NAME]/default/files...
Deploying resources...
Updating deployment state...
Deployment complete!

>>> [CLI] postgres get-synced-table synced_tables/lakebase_test_[UNIQUE_NAME].public.trips_synced
{
"name": "synced_tables/lakebase_test_[UNIQUE_NAME].public.trips_synced",
"unity_catalog_provisioning_state": "ACTIVE"
}

>>> [CLI] bundle summary
Name: deploy-postgres-synced-table-[UNIQUE_NAME]
Target: default
Workspace:
User: [USERNAME]
Path: /Workspace/Users/[USERNAME]/.bundle/deploy-postgres-synced-table-[UNIQUE_NAME]/default
Resources:
Postgres catalogs:
my_catalog:
Name: lakebase_test_[UNIQUE_NAME]
URL: [DATABRICKS_URL]/explore/data/lakebase_test_[UNIQUE_NAME]
Postgres projects:
my_project:
Name: Test Project for Synced Table
URL: (not deployed)
Postgres synced tables:
my_table:
Name: ${resources.postgres_catalogs.my_catalog.catalog_id}.public.trips_synced
URL: [DATABRICKS_URL]/explore/data/$%7Bresources.postgres_catalogs.my_catalog.catalog_id%7D.public.trips_synced
Schemas:
pipeline_storage:
Name: pipeline_storage_[UNIQUE_NAME]
URL: [DATABRICKS_URL]/explore/data/main/pipeline_storage_[UNIQUE_NAME]

>>> print_requests.py --keep --get //postgres ^//workspace-files/ ^//workspace/ ^//telemetry-ext ^//operations/

>>> [CLI] bundle destroy --auto-approve
The following resources will be deleted:
delete resources.postgres_catalogs.my_catalog
delete resources.postgres_projects.my_project
delete resources.postgres_synced_tables.my_table
delete resources.schemas.pipeline_storage

This action will result in the deletion of the following UC schemas. Any underlying data may be lost:
delete resources.schemas.pipeline_storage

This action will result in the deletion of the following Lakebase projects along with
all their branches, databases, and endpoints. All data stored in them will be permanently lost:
delete resources.postgres_projects.my_project

All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/deploy-postgres-synced-table-[UNIQUE_NAME]/default

Deleting files...
Destroy complete!
Cleaning up temporary source table
Loading
Loading