Skip to content

Fix pandas multi-row mixed-type inserts with adaptive casts#68

Open
msrathore-db wants to merge 2 commits into
mainfrom
fix/pecoblr-2746-adaptive-multirow-casts
Open

Fix pandas multi-row mixed-type inserts with adaptive casts#68
msrathore-db wants to merge 2 commits into
mainfrom
fix/pecoblr-2746-adaptive-multirow-casts

Conversation

@msrathore-db
Copy link
Copy Markdown
Collaborator

Summary

  • fix PECOBLR-2746 by adding a targeted insert compiler rewrite that applies CAST(:bind AS <column-type>) only to heterogeneous multi-row bind groups (*_mN) that can trigger Spark inline-table type incompatibility
  • preserve existing behavior for homogeneous multi-row inserts and numeric-only groups (no blanket render-casts)
  • add regression coverage with compiler tests plus an end-to-end pandas to_sql(method='multi') test against a main.default table using mixed object-column values (1, 0, NE)

Test plan

  • poetry run pytest tests/test_local/test_ddl.py -k "AdaptiveMultiRowInsertCasts" -q
  • source ~/.zshrc && export DATABRICKS_SERVER_HOSTNAME="$DATABRICKS_PECOTESTING_SERVER_HOSTNAME" && export DATABRICKS_HTTP_PATH="$DATABRICKS_PECOTESTING_HTTP_PATH" && export DATABRICKS_TOKEN="$DATABRICKS_PECOTESTING_TOKEN" && export DATABRICKS_CATALOG="main" && export DATABRICKS_SCHEMA="default" && poetry run pytest tests/test_local/e2e/test_pandas_multi_mixed_types.py -q

Target cast rendering only for heterogeneous INSERT ... VALUES multi-row bind groups to prevent Spark inline-table type incompatibility, and add compile-time plus e2e tests to validate behavior in main.default.
Replace runtime-value adaptive logic with deterministic casting for SQLAlchemy multi-row VALUES bind markers only, and update compiler tests to verify multi-row casts and single-row non-cast behavior.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant