Migrate converters and Python SDK to flat semantic model documents by flyrain · Pull Request #396 · apache/ossie

flyrain · 2026-09-14T17:47:31Z

Summary

Migrate the Python SDK and remaining affected converters to the single-model document format merged in #383. Ossie input/output now puts name, datasets, relationships, and metrics directly at the root alongside version; the legacy semantic_model wrapper is rejected instead of selecting its first entry.

version: 0.2.0.dev0
name: sales_analytics
datasets:
  - name: orders
    source: sales.public.orders
relationships: []
metrics: []

Rebased onto main at bc7b2c0 after #383 and #407 merged. Microsoft/Power BI migration and its stricter dataset validation are already on main through #407. This PR now contains only the SDK/converter migration and its CI/documentation updates. It also incorporates #397: root-level dialects and vendors are removed; expression dialects and vendor custom extensions remain supported.

Behavior

Flatten the shared OssieDocument API and update dbt, Sigma, Wisdom, Databricks, Honeydew, Omni, OrionBelt, Snowflake, GoodData, NVIDIA GSF, Salesforce, and Polaris readers/writers, fixtures, snapshots, and usage documentation.
Remove obsolete root dialects/vendors fields and first-model/drop-extra-model behavior. Preserve vendor-native collections such as dbt's semantic_models.
Preserve the ontology mapping's embedded semantic_model structure and exclude standalone document metadata from that embedded object.
Add Polaris import --output-dir DIR to emit one document per nonempty namespace with deterministic, collision-safe filenames and no overwrites. Single-file/stdout imports require one nonempty namespace.
Trigger converter CI on shared schema/example changes and run shared SDK tests in Validation CI.

This is a breaking SDK and converter format change for the mutable 0.2.0.dev0 specification. Legacy documents must be migrated first; see the spec's migration guidance.

Checklist

Converter logic, fixtures, examples, and documentation reflect Define one semantic model per document without a wrapper #383.
Breaking behavior and bulk export migration described.
Focused regression coverage added or updated.
Ontology embedding and vendor-specific model collections preserved.
No runtime dependencies added.

Validation

After rebasing onto Fix Microsoft converter for flat documents #407: 43 focused Microsoft tests and 5 focused SDK tests passed. Microsoft lint and whitespace checks passed.
Compared the rebased trees: the Microsoft converter matches main, and the remaining SDK/converter changes are preserved.
Earlier migration validation included 199 focused SDK/converter tests, the Microsoft CLI import/export round-trip, and schema validation of all 10 checked-in standalone converter fixtures. Another 104 focused tests passed for the root-validation fixes.
Full regression suites and optional Microsoft TOM/live-engine checks were not run locally; full regression validation is left to CI.

kayemkim · 2026-09-15T01:25:24Z

#393 was settled by removing the root dialects/vendors arrays from the schema (#397, now on main), and #383 has been asked to drop them in its rebase. This branch still keeps them in a few places that will need the same treatment: converters/sigma/tests/test_roundtrip.py asserts serialized["dialects"] == ["ANSI_SQL", "SIGMA"] and serialized["vendors"] == ["SIGMA"], the orionbelt test_schema_rejects_root_dialects_and_vendors is flipped to test_schema_accepts_root_dialects_and_vendors, the nvidia reader allow-lists both keys at the root, the honeydew tests expect a root vendors, and the SDK tests and python/README.md list them alongside version. I merged the schema change with this branch earlier; the conflict is confined to those spots and resolves by dropping them.

Copilot

🟡 Changes recommended

Unresolved validation gaps and Polaris edge cases block approval.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Migrates the Python SDK and converters to flat semantic-model documents, with Polaris bulk import support and updated tests, documentation, and CI.

Changes:

Flattens OssieDocument and updates converter handling.
Updates fixtures, snapshots, documentation, and validation workflows.
Adds deterministic Polaris namespace output support.

File summaries

File	Reviewed change
`python/tests/test_models.py`	Flat model and legacy-shape tests
`python/src/ossie/models.py`	Flat document model
`python/README.md`	SDK format documentation
`docs/index.md`	Format overview updates
`converters/wisdom/tests/test_wisdom_to_ossie.py`	Wisdom conversion tests
`converters/wisdom/tests/test_ossie_to_wisdom.py`	Wisdom reverse-conversion tests
`converters/wisdom/src/ossie_wisdom/wisdom_to_ossie.py`	Wisdom flat-document output
`converters/wisdom/src/ossie_wisdom/ossie_to_wisdom.py`	Wisdom flat-document input
`converters/wisdom/src/ossie_wisdom/converter_issues.py`	Wisdom conversion issues
`converters/wisdom/src/ossie_wisdom/cli.py`	Wisdom CLI handling
`converters/wisdom/README.md`	Wisdom documentation
`converters/snowflake/tests/test_ossie_to_snowflake_yaml_converter.py`	Snowflake conversion tests
`converters/snowflake/src/ossie_snowflake/converter.py`	Snowflake flat input conversion
`converters/snowflake/README.md`	Snowflake documentation
`converters/sigma/tests/test_sigma_to_ossie.py`	Sigma import tests
`converters/sigma/tests/test_roundtrip.py`	Sigma round-trip tests
`converters/sigma/tests/test_ossie_to_sigma.py`	Sigma export tests
`converters/sigma/src/ossie_sigma/sigma_to_ossie.py`	Sigma flat-document import
`converters/sigma/src/ossie_sigma/ossie_to_sigma.py`	Sigma flat-document export
`converters/sigma/src/ossie_sigma/converter_issues.py`	Sigma conversion issues
`converters/sigma/README.md`	Sigma documentation
`converters/sigma/LIMITATIONS.md`	Sigma limitations
`converters/salesforce/src/test/resources/schemas/salesforce-input-fixture-schema.json`	Salesforce fixture schema
`converters/salesforce/src/test/java/org/apache/ossie/SalesforceToOssieConverterTest.java`	Salesforce conversion tests
`converters/salesforce/src/test/java/org/apache/ossie/converter/FlatDocumentConversionTest.java`	Flat-document conversion tests
`converters/salesforce/src/main/java/org/apache/ossie/converter/ConverterImpl.java`	Salesforce conversion implementation
`converters/salesforce/src/main/java/org/apache/ossie/converter/ConverterConstants.java`	Salesforce converter constants
`converters/salesforce/src/main/java/org/apache/ossie/converter/Converter.java`	Salesforce converter interface
`converters/salesforce/README.md`	Salesforce documentation
`converters/README.md`	Converter documentation
`converters/polaris/src/main/java/org/apache/ossie/converter/polaris/PolarisImporter.java`	Polaris import handling
`converters/polaris/src/main/java/org/apache/ossie/converter/polaris/PolarisExporter.java`	Polaris export handling
`converters/polaris/src/main/java/org/apache/ossie/converter/polaris/OssieYamlGenerator.java`	Ossie YAML generation
`converters/polaris/src/main/java/org/apache/ossie/converter/polaris/OssiePolarisConverter.java`	Polaris conversion and bulk output
`converters/polaris/src/main/java/org/apache/ossie/converter/polaris/OssieModelParser.java`	Flat YAML parsing
`converters/polaris/src/main/java/org/apache/ossie/converter/polaris/model/OssieModel.java`	Polaris Ossie model
`converters/polaris/README.md`	Polaris documentation
`converters/orionbelt/tests/test_ossie_metric_no_silent_loss.py`	Metric preservation tests
`converters/orionbelt/tests/test_ossie_converter_vendors.py`	Vendor conversion tests
`converters/orionbelt/tests/test_ossie_converter_trend_v26.py`	Trend conversion tests
`converters/orionbelt/tests/test_ossie_converter_roundtrip_robustness.py`	Round-trip robustness tests
`converters/orionbelt/tests/test_ossie_converter_pop.py`	Population conversion tests
`converters/orionbelt/tests/test_ossie_converter_ontology.py`	Ontology conversion tests
`converters/orionbelt/tests/test_ossie_converter_measure_overrides.py`	Measure override tests
`converters/orionbelt/tests/test_ossie_converter_filters.py`	Filter conversion tests
`converters/orionbelt/tests/test_ossie_converter_cumulative.py`	Cumulative conversion tests
`converters/orionbelt/src/ossie_orionbelt/validation.py`	OrionBelt validation
`converters/orionbelt/src/ossie_orionbelt/ossie_to_obml.py`	Ossie-to-OBML conversion
`converters/orionbelt/src/ossie_orionbelt/ontology.py`	OrionBelt ontology handling
`converters/orionbelt/src/ossie_orionbelt/obml_to_ossie.py`	OBML-to-Ossie conversion
`converters/orionbelt/README.md`	OrionBelt documentation
`converters/orionbelt/ossie_obml_ontology_mapping_analysis.md`	Ontology mapping analysis
`converters/orionbelt/ossie_obml_mapping_analysis.md`	OBML mapping analysis
`converters/ontology/src/ossie_ontology/spec.py`	Ontology specification
`converters/omni/tests/test_real_world_layout.py`	Omni layout tests
`converters/omni/tests/test_ossie_to_omni.py`	Omni export tests
`converters/omni/tests/test_omni_to_ossie.py`	Omni import tests
`converters/omni/tests/fixtures/fixtureA_ossie.yaml`	Omni fixture
`converters/omni/tests/_util.py`	Omni test utilities
`converters/omni/tests/_roundtrip_helpers.py`	Omni round-trip helpers
`converters/omni/src/ossie_omni/ossie_to_omni.py`	Omni flat-document export
`converters/omni/src/ossie_omni/omni_to_ossie.py`	Omni flat-document import
`converters/omni/README.md`	Omni documentation
`converters/nvidia/tests/test_converter.py`	NVIDIA converter tests
`converters/nvidia/tests/fixtures/sales.ossie.yaml`	NVIDIA fixture
`converters/nvidia/src/ossie_nvidia_gsf/native_converter.py`	NVIDIA native conversion
`converters/nvidia/README.md`	NVIDIA documentation
`converters/microsoft/tests/test_tom_integration.py`	Microsoft TOM integration tests
`converters/microsoft/tests/test_semantic_model_to_ossie.py`	Microsoft import tests
`converters/microsoft/tests/test_ossie_to_semantic_model.py`	Microsoft export tests
`converters/microsoft/tests/test_edge_cases.py`	Microsoft edge-case tests
`converters/microsoft/tests/conftest.py`	Microsoft test configuration
`converters/microsoft/src/ossie_microsoft/semantic_model_to_ossie.py`	Microsoft flat-document import
`converters/microsoft/src/ossie_microsoft/ossie_to_semantic_model.py`	Microsoft flat-document export
`converters/microsoft/README.md`	Microsoft documentation
`converters/honeydew/tests/test_ossie_honeydew_converter.py`	Honeydew converter tests
`converters/honeydew/src/ossie_honeydew/converter.py`	Honeydew conversion
`converters/honeydew/README.md`	Honeydew documentation
`converters/gooddata/tests/test_roundtrip.py`	GoodData round-trip tests
`converters/gooddata/tests/test_ossie_to_gooddata.py`	GoodData export tests
`converters/gooddata/tests/test_gooddata_to_ossie.py`	GoodData import tests
`converters/gooddata/src/ossie_gooddata/ossie_to_gooddata.py`	GoodData flat-document export
`converters/gooddata/src/ossie_gooddata/models.py`	GoodData models
`converters/gooddata/src/ossie_gooddata/gooddata_to_ossie.py`	GoodData flat-document import
`converters/gooddata/README.md`	GoodData documentation
`converters/dbt/tests/test_ossie_to_msi.py`	dbt export tests
`converters/dbt/tests/test_msi_to_ossie.py`	dbt import tests
`converters/dbt/tests/helpers.py`	dbt test helpers
`converters/dbt/tests/__snapshots__/test_ossie_to_msi.ambr`	dbt export snapshots
`converters/dbt/tests/__snapshots__/test_msi_to_ossie.ambr`	dbt import snapshots
`converters/dbt/src/ossie_dbt/ossie_to_msi.py`	dbt flat-document export
`converters/dbt/src/ossie_dbt/msi_to_ossie.py`	dbt flat-document import
`converters/dbt/README.md`	dbt documentation
`converters/databricks/tests/test_metric_view_to_ossie.py`	Databricks import tests
`converters/databricks/tests/fixtures/tpcds_ossie.yaml`	Databricks TPC-DS fixture
`converters/databricks/tests/fixtures/fixtureB_ossie.yaml`	Databricks fixture
`converters/databricks/tests/fixtures/fixtureA_ossie.yaml`	Databricks fixture
`converters/databricks/tests/_util.py`	Databricks test utilities
`converters/databricks/tests/_roundtrip_helpers.py`	Databricks round-trip helpers
`converters/databricks/src/ossie_databricks/ossie_to_metric_view.py`	Databricks export
`converters/databricks/src/ossie_databricks/metric_view_to_ossie.py`	Databricks import
`converters/databricks/README.md`	Databricks documentation
`.github/workflows/validation-ci.yml`	Shared validation workflow
`.github/workflows/converter-wisdom-ci.yml`	Wisdom CI workflow
`.github/workflows/converter-snowflake-ci.yml`	Snowflake CI workflow
`.github/workflows/converter-sigma-ci.yml`	Sigma CI workflow
`.github/workflows/converter-salesforce-ci.yml`	Salesforce CI workflow
`.github/workflows/converter-polaris-ci.yml`	Polaris CI workflow
`.github/workflows/converter-orionbelt-ci.yml`	OrionBelt CI workflow
`.github/workflows/converter-ontology-ci.yml`	Ontology CI workflow
`.github/workflows/converter-omni-ci.yml`	Omni CI workflow
`.github/workflows/converter-nvidia-ci.yml`	NVIDIA CI workflow
`.github/workflows/converter-microsoft-ci.yml`	Microsoft CI workflow
`.github/workflows/converter-honeydew-ci.yml`	Honeydew CI workflow
`.github/workflows/converter-gooddata-ci.yml`	GoodData CI workflow
`.github/workflows/converter-dbt-ci.yml`	dbt CI workflow
`.github/workflows/converter-databricks-ci.yml`	Databricks CI workflow

Review details

Suppressed comments (3)

converters/orionbelt/src/ossie_orionbelt/obml_to_ossie.py:146

This comment says root-level dialect/vendor advertisement arrays remain optional, but the migrated schema removes those properties (additionalProperties: false) and the converter above rejects them. Please describe them as unsupported rather than optional so the implementation contract is not contradicted.
converters/polaris/src/main/java/org/apache/ossie/converter/polaris/OssiePolarisConverter.java:148
The new --output-dir path is skipped when the catalog has no nonempty namespaces because this early return also handles single-file and stdout imports. The documented contract says those modes require exactly one nonempty namespace, so an empty catalog should fail instead of returning success without producing output.
converters/polaris/src/main/java/org/apache/ossie/converter/polaris/OssiePolarisConverter.java:164
The new bulk-import path writes the generator output without constraining catalog identifiers, but OssieYamlGenerator emits names and other identifiers as unquoted scalars. A valid Polaris namespace such as true becomes name: true, which SnakeYAML reads as a boolean and OssieModelParser then rejects, leaving an emitted file that cannot be re-imported. Quote all generated string fields (or use a YAML serializer) before writing these documents.

Files reviewed: 125/125 changed files
Comments generated: 6
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

jbonofre · 2026-09-16T17:58:58Z

I would appreciate to review this PR as I'm doing a bunch of work on converters.

Follows apache#383, which moved the semantic model's fields to the document root and removed the `semantic_model` wrapper. `ossie-schema.json` now requires `version`, `name` and `datasets` at the root and forbids additional properties, so the converter's emitted documents stopped validating the moment the branch was rebased. Emit side is `{"version": ..., **semantic_model}`. The spread cannot clobber `version`: the model dict only ever holds name/description/datasets/ relationships/metrics plus the `custom_extensions` the stash writes, and no path on it emits a `version` key. Read side rejects the old wrapper by name rather than lifting its first entry, matching the treatment in apache#396. Reading entry zero would convert silently while discarding any later model, and a document old enough to carry the wrapper may have moved on elsewhere too; naming the one thing the reader must change is more useful than a best-effort guess. `datasets` is `minItems: 1` upstream, so the emit side now reports TS-MODEL-NO-DATASETS at ERROR when a model yields none. Without it the two legs disagreed: `to-ossie` exited 0 having written a document that fails the schema and that this converter's own `to-tml` refuses. Both legs now exit 1 on the same input. Both fixtures were regenerated by running the migrated converter over their own TML and asserting the result deep-equals the old expected document with its wrapper lifted -- so the 1,151 changed fixture lines are a re-rooting, not a content change. 823 tests pass. The README's mapping row is now pinned by a test, having been verified by nothing: reverting it left the whole suite green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

kayemkim

Ran this against the validation and all thirteen converter workflows locally on the rebased head (366572c on bc7b2c0). Everything I could run is green and matches the checks here, and the two Java suites are green in CI. A few things beyond the test suites:

The repository's own documents now validate. On main, 10 of the 11 Ossie documents under examples/ and converters/** fail against the schema because the fixtures still carry the wrapper. On this branch all pass except the salesforce example, which fails only on its four pre-existing [SQL] lines (bracketed identifiers), the same as before #383.

The wrapper is rejected explicitly rather than mis-read. Feeding the pre-#383 TPC-DS example to the SDK gives semantic_model: Extra inputs are not permitted alongside the missing name/datasets, and the sigma, snowflake, honeydew and nvidia CLIs all exit 1 with a message naming the wrapper. The flat example converts on all four. The five places I listed on the 15th are all handled, and 366572c gives every converter the same Root dialects and vendors are not supported check, which also covers Copilot's six inline notes on the earlier head.

The core-spec/** and examples/** triggers on the converter workflows are welcome. That gap is what let #297 sit with a failing orionbelt suite for a week and let #383 land with the Microsoft suite broken until #407.

For whoever merges, I merged each open converter-area PR onto this branch. New conflicts this PR introduces, each one file: #404 (salesforce README), #378 (dbt msi_to_ossie.py), #333 (databricks README), #338 (python/tests/test_models.py). Merging cleanly but still reading the wrapper, so they will break at runtime once this lands: the new converters in #289, #364, #352, #360, #381, #320, the interop harness in #348, and the Java converter in #333. Their authors will need a heads-up since neither the schema nor the SDK gate catches a parser that looks for semantic_model itself.

I have not gone through the Polaris --output-dir Java changes line by line. It follows from one model per document, and its tests pass. LGTM for the migration.

flyrain · 2026-09-18T17:04:34Z

Hi @khush-bhatia @jbonofre, could you take a look at this PR? Don't want the main branch in a inconsistent status for too long. Thanks!

flyrain mentioned this pull request Sep 14, 2026

Define one semantic model per document without a wrapper #383

Merged

7 tasks

kayemkim mentioned this pull request Sep 15, 2026

fix(core-spec): drop root-level dialects and vendors from the schema #397

Merged

17 tasks

flyrain force-pushed the yf/migrate-converters-flat-model branch from b2bfb27 to 9c2a506 Compare September 16, 2026 17:27

flyrain marked this pull request as ready for review September 16, 2026 17:33

Copilot AI lite review requested due to automatic review settings September 16, 2026 17:33

Copilot started reviewing on behalf of flyrain September 16, 2026 17:40 View session

Copilot AI reviewed Sep 16, 2026

View reviewed changes

jbonofre self-requested a review September 16, 2026 17:57

eisber mentioned this pull request Sep 16, 2026

Fix Microsoft converter for flat documents #407

Merged

djwaldo mentioned this pull request Sep 17, 2026

feat(thoughtspot): bidirectional ThoughtSpot TML <-> Ossie converter #364

Open

flyrain added 8 commits September 17, 2026 13:40

Migrate GoodData and NVIDIA converters to flat model documents

53e06b7

Migrate Python SDK and dbt Sigma Wisdom converters to flat documents

81d7b4d

Migrate five Python converters to flat Ossie documents

430db01

Migrate Java converters to flat single-model documents

a807478

Fix Honeydew roundtrip expectation for document metadata

4aa875e

Align converters with merged flat spec and migrate Microsoft

ba0dade

Fix dbt dialect tests for flat document metadata

a427fd3

Validate flat document roots consistently across converters

366572c

flyrain force-pushed the yf/migrate-converters-flat-model branch from ad8f337 to 366572c Compare September 17, 2026 20:41

kayemkim mentioned this pull request Sep 17, 2026

chore: add missing ASF license headers #416

Merged

17 tasks

kayemkim reviewed Sep 17, 2026

View reviewed changes

eisber mentioned this pull request Sep 18, 2026

Update remaining converters for the flat (one-model-per-document) Ossie schema #418

Open

Navigation Menu

Migrate converters and Python SDK to flat semantic model documents - #396

Migrate converters and Python SDK to flat semantic model documents#396
flyrain wants to merge 8 commits into
apache:mainfrom
flyrain:yf/migrate-converters-flat-model

flyrain commented Sep 14, 2026 •

edited

Loading

Uh oh!

kayemkim commented Sep 15, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jbonofre commented Sep 16, 2026

Uh oh!

kayemkim left a comment

Uh oh!

flyrain commented Sep 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

flyrain commented Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Behavior

Checklist

Validation

Uh oh!

kayemkim commented Sep 15, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Suppressed comments (3)

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jbonofre commented Sep 16, 2026

Uh oh!

kayemkim left a comment

Choose a reason for hiding this comment

Uh oh!

flyrain commented Sep 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

flyrain commented Sep 14, 2026 •

edited

Loading