Repository navigation
Conversation
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Eric Sibly <eric.sibly@avanade.com>
Extract the format-neutral coordinator, processing modes, diagnostics, metadata, scoped execution, and replay. Migrate CSV, File CodeGen, consumers, tests, and generated configuration references; simplify extension hooks with focused argument objects. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Eric Sibly <eric.sibly@avanade.com>
Deliver the Phase 2 JSONL runtime, metadata routing, nested mapping and File CodeGen with generated integration consumers. Simplify parsing and policies, reduce the public API, and share cardinality and record-position enums across formats. Update schema, documentation, AI guidance, and focused coverage. Template-pack validation remains explicitly deferred. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Eric Sibly <eric.sibly@avanade.com>
Use System.Text.Json for JSON object deserialization, generate collection validation, split IO documentation by namespace, and demonstrate simple and complex CSV, JSON, and JSONL Blob imports. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Eric Sibly <eric.sibly@avanade.com>
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The Ignore policy documentation promises UTF-8 validation of unknown values, but tests confirm ignored strings are not decoded and may contain malformed UTF-8.
1 open issue
What changed in this PR
This PR extends CoreEx.IO’s CSV ingestion foundation to JSONL and streaming JSON documents, sharing validation, reporting, replay, and scoped execution across formats.
Changes:
- Introduces shared processing contracts, metadata rules, telemetry, and JSON configuration.
- Extends File CodeGen and generated examples for nested records, collections, and metadata.
- Adds multi-format inventory import examples, fixtures, and Functions runtime wiring.
| File | Description |
|---|---|
| tests/CoreEx.IO.Test.Unit/GlobalUsing.cs | Imports JSON runtimes and generated fixtures. |
| tests/CoreEx.IO.Test.Unit/Files/RoutedJsonl/RoutedJsonlTrailerValidator.g.cs | Validates trailer totals. |
| tests/CoreEx.IO.Test.Unit/Files/RoutedJsonl/RoutedJsonlTrailerMapping.g.cs | Maps trailer fields. |
| tests/CoreEx.IO.Test.Unit/Files/RoutedJsonl/RoutedJsonlTrailer.g.cs | Defines trailer wire data. |
| tests/CoreEx.IO.Test.Unit/Files/RoutedJsonl/RoutedJsonlOrderValidator.g.cs | Validates routed orders. |
| tests/CoreEx.IO.Test.Unit/Files/RoutedJsonl/RoutedJsonlOrderMapping.g.cs | Maps routed order fields. |
| tests/CoreEx.IO.Test.Unit/Files/RoutedJsonl/RoutedJsonlOrder.g.cs | Defines routed order data. |
| tests/CoreEx.IO.Test.Unit/Files/RoutedJsonl/RoutedJsonlHeaderValidator.g.cs | Validates batch identifiers. |
| tests/CoreEx.IO.Test.Unit/Files/RoutedJsonl/RoutedJsonlHeaderMapping.g.cs | Maps header fields. |
| tests/CoreEx.IO.Test.Unit/Files/RoutedJsonl/RoutedJsonlHeader.g.cs | Defines header wire data. |
| tests/CoreEx.IO.Test.Unit/Files/RoutedJsonl/RoutedJsonlHandler.cs | Adds a scoped handler starter. |
| tests/CoreEx.IO.Test.Unit/Files/OrdersJsonl/OrdersJsonlLineValidator.g.cs | Validates nested order lines. |
| tests/CoreEx.IO.Test.Unit/Files/OrdersJsonl/OrdersJsonlLineMapping.g.cs | Maps nested line fields. |
| tests/CoreEx.IO.Test.Unit/Files/OrdersJsonl/OrdersJsonlLine.g.cs | Defines order-line wire data. |
| tests/CoreEx.IO.Test.Unit/Files/OrdersJsonl/OrdersJsonlHandler.cs | Delegates imports to the application. |
| tests/CoreEx.IO.Test.Unit/Files/OrdersJsonl/OrdersJsonlContext.g.cs | Adds pass-local JSONL context. |
| tests/CoreEx.IO.Test.Unit/Files/OrdersJsonl/OrdersJsonlAddressValidator.g.cs | Validates nested addresses. |
| tests/CoreEx.IO.Test.Unit/Files/OrdersJsonl/OrdersJsonlAddressMapping.g.cs | Maps address fields. |
| tests/CoreEx.IO.Test.Unit/Files/OrdersJsonl/OrdersJsonlAddress.g.cs | Defines address wire data. |
| tests/CoreEx.IO.Test.Unit/Files/OrdersJsonl/IOrderImportApplication.cs | Defines application-owned execution. |
| tests/CoreEx.IO.Test.Unit/FileBaselineFixtureTests.cs | Uses the renamed content-failure outcome. |
| tests/CoreEx.IO.Test.Unit/CoreEx.IO.Test.Unit.csproj | Signs the test assembly. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/JsonlFiles/InventoryJsonlHandler.cs | Delegates JSONL adjustments to the service. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/JsonlFiles/InventoryJsonlContext.g.cs | Defaults JSONL imports to validated replay. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/JsonlFiles/InventoryJsonlAdjustmentMapping.g.cs | Maps JSONL adjustments. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/JsonlFiles/InventoryJsonlAdjustment.g.cs | Defines JSONL adjustment data. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/JsonlBatchFiles/InventoryJsonlBatchTrailerMapping.g.cs | Maps JSONL reconciliation totals. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/JsonlBatchFiles/InventoryJsonlBatchTrailer.g.cs | Defines JSONL trailer data. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/JsonlBatchFiles/InventoryJsonlBatchLineMapping.g.cs | Maps nested JSONL bin quantities. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/JsonlBatchFiles/InventoryJsonlBatchLine.g.cs | Defines JSONL bin-line data. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/JsonlBatchFiles/InventoryJsonlBatchHeaderValidator.g.cs | Validates JSONL batch identifiers. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/JsonlBatchFiles/InventoryJsonlBatchHeaderMapping.g.cs | Maps JSONL batch headers. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/JsonlBatchFiles/InventoryJsonlBatchHeader.g.cs | Defines JSONL header data. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/JsonlBatchFiles/InventoryJsonlBatchHandler.cs | Validates and sums nested quantities. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/JsonlBatchFiles/InventoryJsonlBatchAdjustmentMapping.g.cs | Maps JSONL adjustments and children. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/JsonlBatchFiles/InventoryJsonlBatchAdjustment.g.cs | Defines nested JSONL adjustment data. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/JsonFiles/InventoryJsonHandler.cs | Delegates JSON adjustments to the service. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/JsonFiles/InventoryJsonContext.g.cs | Defaults JSON imports to validated replay. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/JsonFiles/InventoryJsonAdjustmentMapping.g.cs | Maps JSON adjustment fields. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/JsonFiles/InventoryJsonAdjustment.g.cs | Defines JSON adjustment data. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/JsonBatchFiles/InventoryJsonBatchTrailerValidator.g.cs | Validates JSON trailer totals. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/JsonBatchFiles/InventoryJsonBatchTrailerMapping.g.cs | Maps JSON trailer totals. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/JsonBatchFiles/InventoryJsonBatchTrailer.g.cs | Defines JSON trailer data. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/JsonBatchFiles/InventoryJsonBatchLineValidator.g.cs | Validates JSON bin lines. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/JsonBatchFiles/InventoryJsonBatchLineMapping.g.cs | Maps JSON bin quantities. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/JsonBatchFiles/InventoryJsonBatchLine.g.cs | Defines JSON bin-line data. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/JsonBatchFiles/InventoryJsonBatchHeaderValidator.g.cs | Validates JSON batch identifiers. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/JsonBatchFiles/InventoryJsonBatchHeaderMapping.g.cs | Maps JSON batch headers. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/JsonBatchFiles/InventoryJsonBatchHeader.g.cs | Defines JSON header data. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/JsonBatchFiles/InventoryJsonBatchHandler.cs | Validates and sums nested quantities. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/JsonBatchFiles/InventoryJsonBatchAdjustmentMapping.g.cs | Maps JSON adjustments and children. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/JsonBatchFiles/InventoryJsonBatchAdjustment.g.cs | Defines nested JSON adjustment data. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/InventoryJsonProcessor.cs | Configures tenant context and reporting. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/InventoryJsonlProcessor.cs | Configures tenant context and reporting. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/InventoryJsonlBatchContext.cs | Holds pass-local JSONL totals. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/InventoryJsonBatchContext.cs | Holds pass-local JSON totals. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/InventoryImportReport.cs | Shares diagnostic report serialization. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/InventoryCsvBatchContext.cs | Holds pass-local CSV totals. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/Files/InventoryHandler.cs | Adopts shared execution arguments. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/Files/InventoryAdjustmentMapping.g.cs | Generalizes mapping documentation only. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/example.jsonl | Adds simple JSONL input. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/example.json | Adds root-array JSON input. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/example-batch.jsonl | Adds routed JSONL batch input. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/example-batch.json | Adds enveloped JSON batch input. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/example-batch.csv | Adds hierarchical CSV batch input. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/CsvBatchFiles/InventoryCsvBatchTrailerValidator.g.cs | Validates CSV trailer totals. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/CsvBatchFiles/InventoryCsvBatchTrailerMapping.g.cs | Maps positional CSV totals. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/CsvBatchFiles/InventoryCsvBatchTrailer.g.cs | Defines CSV trailer data. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/CsvBatchFiles/InventoryCsvBatchLineValidator.g.cs | Validates CSV bin lines. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/CsvBatchFiles/InventoryCsvBatchLineMapping.g.cs | Maps positional CSV bin fields. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/CsvBatchFiles/InventoryCsvBatchLine.g.cs | Defines CSV bin-line data. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/CsvBatchFiles/InventoryCsvBatchHeaderValidator.g.cs | Validates CSV batch identifiers. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/CsvBatchFiles/InventoryCsvBatchHeaderMapping.g.cs | Maps CSV batch headers. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/CsvBatchFiles/InventoryCsvBatchHeader.g.cs | Defines CSV header data. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/CsvBatchFiles/InventoryCsvBatchHandler.cs | Validates and executes grouped adjustments. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/CsvBatchFiles/InventoryCsvBatchAdjustmentValidator.g.cs | Validates adjustment identifiers. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/CsvBatchFiles/InventoryCsvBatchAdjustmentMapping.g.cs | Maps positional adjustment fields. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/CsvBatchFiles/InventoryCsvBatchAdjustmentGroup.g.cs | Groups adjustments with child lines. |
| tests/CoreEx.Azure.Functions.TestApp/InventoryImports/CsvBatchFiles/InventoryCsvBatchAdjustment.g.cs | Defines CSV adjustment data. |
| tests/CoreEx.Azure.Functions.TestApp/GlobalUsing.cs | Imports JSON processing namespaces. |
| tests/CoreEx.Azure.Functions.TestApp/CoreEx.Azure.Functions.TestApp.csproj | Embeds example inputs. |
| tests/CoreEx.Azure.Functions.TestApp.CodeGen/inventory-jsonl-file.yaml | Defines simple JSONL generation. |
| tests/CoreEx.Azure.Functions.TestApp.CodeGen/inventory-json-file.yaml | Defines simple JSON generation. |
| tests/CoreEx.Azure.Functions.TestApp.CodeGen/inventory-file.yaml | Migrates CSV configuration syntax. |
| tests/aspire/CoreEx.Azure.Functions.Test.Aspire/BlobRuntimeTests.cs | Waits for new import functions. |
| tests/aspire/CoreEx.Azure.Functions.Test.AppHost/AppHost.cs | Enables new import functions. |
| src/CoreEx.IO/JsonLines/JsonLinesFrame.cs | Tracks line bytes and provenance. |
| src/CoreEx.IO/JsonLines/JsonLinesFileOptions.cs | Configures JSONL framing and routing. |
| src/CoreEx.IO/JsonLines/JsonLinesBlankLinePolicy.cs | Defines blank-line handling. |
| src/CoreEx.IO/Json/JsonUnknownMemberPolicy.cs | Defines unknown-member handling. |
| src/CoreEx.IO/Json/JsonFileOptions.cs | Configures member policy and depth. |
| src/CoreEx.IO/Json/JsonFileContext.cs | Exposes document reconciliation context. |
| src/CoreEx.IO/Json/JsonDocumentToken.cs | Carries token source spans. |
| src/CoreEx.IO/Json/JsonDocumentPosition.cs | Tracks paths and duplicate status. |
| src/CoreEx.IO/Json/JsonDocumentKind.cs | Defines array/object unit selection. |
| src/CoreEx.IO/GlobalUsing.cs | Imports System.Text.Json APIs. |
| src/CoreEx.IO/FileValidationArgs.cs | Supplies EOF validation arguments. |
| src/CoreEx.IO/FileSummaryArgs.cs | Supplies finalization arguments. |
| src/CoreEx.IO/FileRecordPosition.cs | Shares source-position restrictions. |
| src/CoreEx.IO/FileProcessingTelemetry.cs | Adds format-neutral metrics. |
| src/CoreEx.IO/FileProcessingOutcome.cs | Simplifies returned processing outcomes. |
| src/CoreEx.IO/FileProcessingMode.cs | Adds single-pass validation/execution. |
| src/CoreEx.IO/FileExecutionFailedArgs.cs | Carries business failure provenance. |
| src/CoreEx.IO/FileDiagnosticCategory.cs | Removes the processing-limit category. |
| src/CoreEx.IO/FileDiagnosticArgs.cs | Supplies shared diagnostic arguments. |
| src/CoreEx.IO/FileDateTime.cs | Shares explicit-offset detection. |
| src/CoreEx.IO/FileCardinality.cs | Shares occurrence-count rules. |
| src/CoreEx.IO/Csv/CsvRecordPosition.cs | Removes the CSV-specific position enum. |
| src/CoreEx.IO/Csv/CsvRecordCardinality.cs | Removes the CSV-specific cardinality enum. |
| src/CoreEx.IO/Csv/CsvProcessingTelemetry.cs | Removes CSV-specific metrics. |
| src/CoreEx.IO/Csv/CsvParsedRecord.cs | Updates terminal-condition documentation only. |
| src/CoreEx.IO/Csv/CsvMetadataDefinition.cs | Defines CSV metadata routes. |
| src/CoreEx.IO/Csv/CsvHierarchyNode.cs | Adopts shared cardinality. |
| src/CoreEx.IO/Csv/CsvFileMapping.cs | Makes header detection internal. |
| src/CoreEx.IO/CoreEx.IO.csproj | Updates package description and test access. |
| src/CoreEx.CodeGen/Scripts/file-script.yaml | Generalizes mapping generation labels. |
| src/CoreEx.CodeGen/Files/Templates/FileValidator_cs.hbs | Uses resolved CLR property names. |
| src/CoreEx.CodeGen/Files/Templates/FileRecord_cs.hbs | Uses resolved CLR property names. |
| src/CoreEx.CodeGen/Files/docs/Unit.md | Documents business-unit selection. |
| src/CoreEx.CodeGen/Files/docs/Structure.md | Removes obsolete envelope options. |
| src/CoreEx.CodeGen/Files/docs/Metadata.md | Documents shared metadata configuration. |
| src/CoreEx.CodeGen/Files/docs/JsonlRecord.md | Documents discriminator matching. |
| src/CoreEx.CodeGen/Files/docs/JsonlMetadata.md | Documents JSONL metadata placement. |
| src/CoreEx.CodeGen/Files/docs/Jsonl.md | Documents framing and routing options. |
| src/CoreEx.CodeGen/Files/docs/JsonDocumentMetadata.md | Documents metadata pointer selection. |
| src/CoreEx.CodeGen/Files/docs/Json.md | Documents shared JSON settings. |
| src/CoreEx.CodeGen/Files/docs/CsvRecord.md | Documents CSV route constraints. |
| src/CoreEx.CodeGen/Files/docs/CsvProperty.md | Documents positional field selection. |
| src/CoreEx.CodeGen/Files/docs/CsvMetadata.md | Documents CSV metadata placement. |
| src/CoreEx.CodeGen/Files/docs/Collection.md | Documents nested collection constraints. |
| src/CoreEx.CodeGen/Files/Config/NodeConfig.cs | Emits shared cardinality references. |
| src/CoreEx.CodeGen/Files/Config/JsonLinesRecordConfig.cs | Validates explicit JSONL route matches. |
| src/CoreEx.CodeGen/Files/Config/JsonLinesMetadataConfig.cs | Validates JSONL metadata placement. |
| src/CoreEx.CodeGen/Files/Config/JsonDocumentMetadataConfig.cs | Configures JSON metadata pointers. |
| src/CoreEx.CodeGen/Files/Config/CsvPropertyConfig.cs | Validates positional CSV selectors. |
| src/CoreEx.CodeGen/Files/Config/CsvMetadataConfig.cs | Validates CSV metadata placement. |
| docs/design/io-multiformat/phase0-fixtures/valid.jsonl | Adds valid nested JSONL fixture. |
| docs/design/io-multiformat/phase0-fixtures/valid.csv | Adds equivalent hierarchical CSV fixture. |
| docs/design/io-multiformat/phase0-fixtures/valid-envelope.json | Adds JSON envelope fixture. |
| docs/design/io-multiformat/phase0-fixtures/valid-array.json | Adds JSON root-array fixture. |
| docs/design/io-multiformat/phase0-fixtures/trailing-garbage.json | Adds trailing-content failure fixture. |
| docs/design/io-multiformat/phase0-fixtures/single-object.json | Adds root-object fixture. |
| docs/design/io-multiformat/phase0-fixtures/routed.jsonl | Adds metadata-routed JSONL fixture. |
| docs/design/io-multiformat/phase0-fixtures/missing-child.jsonl | Adds missing-collection fixture. |
| docs/design/io-multiformat/phase0-fixtures/malformed-line.jsonl | Adds malformed-line fixture. |
| docs/design/io-multiformat/phase0-fixtures/late-total.json | Adds reconciliation-mismatch fixture. |
| docs/design/io-multiformat/phase0-fixtures/invalid-child.jsonl | Adds nested-validation failure fixture. |
| docs/design/io-multiformat/phase0-fixtures/empty-array.json | Adds empty-array fixture. |
| AGENTS.md | Expands package descriptions to JSONL. |
| .github/prompts/coreex-file.prompt.md | Expands prompt scope to JSONL. |
| .github/coreex-ai-workflows.md | Expands file-workflow description. |
| .claude/commands/coreex-file.md | Expands command scope to JSONL. |
Review effort: Balanced
Clarify ignored JSON string validation, use the streaming-capable serializer on .NET 8, and read runtime report progress without deserializing internal bookkeeping. Include migration notes, packaged documentation, generated-name guards, and expanded CI consumer checks. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Eric Sibly <eric.sibly@avanade.com>
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
Intentional public API, configuration, and telemetry breaks require maintainer compatibility review, while supported-framework CI and template-pack/scaffold validation remain pending.
No open issues 🎉
1 resolved since last review
Review effort: Balanced
Assert omitted zero-valued report counters using the actual serialized report contract and cover late content failure in component tests. Upgrade Aspire hosting and WebUtilities dependency floors, including sample hosts and generated Aspire projects, instead of suppressing NuGet audit errors. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Eric Sibly <eric.sibly@avanade.com>
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
Intentional public API/configuration breaks and packaged template dependency changes require maintainer compatibility review and supported-framework/template validation that remain pending and cannot be established from this source-only review.
No open issues 🎉
Review effort: Balanced
Use explicit None items so multi-target pack evaluation includes the migration and namespace guides. Verify all packed guides and supported assemblies, plus seven packaged AI/Aspire/JSON consumer scenarios. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Eric Sibly <eric.sibly@avanade.com>
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
Public API and configuration breaking changes require maintainer compatibility review, while SDK and dependency upgrades still await supported-framework CI and template-pack/scaffold validation.
No open issues 🎉
Review effort: Balanced
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Eric Sibly <eric.sibly@avanade.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Extend CoreEx.IO from CSV-only ingestion to CSV, JSONL, and streaming JSON documents, with a shared processing lifecycle and generation-first integrations. Includes the multi-format foundation through Phase 3 and the Phase 4 API/migration, framework, packaging, and release-note sweep. Intentional compatibility breaks still require maintainer acceptance before merge.
Compatibility and operational boundaries
This includes intentional breaking changes to the file-processing API, CSV YAML configuration, shared cardinality/position types, and telemetry. See the 4.3-to-4.4 migration guide.
CoreEx.IO and CoreEx.CodeGen retain .NET 8, 9, and 10 targets. CoreEx.IO's .NET 8 package requires System.Text.Json 9.0.21 or later to avoid version 8 buffering entire ignored containers during async deserialization. The native Functions examples target .NET 10. Ignore skips decoding unknown string values and therefore does not guarantee their UTF-8 validity; structural checks remain enabled.
ValidateThenExecute preserves whole-file acceptance before replay; ValidateAndExecute explicitly permits partial execution. Source admission, input size, concurrency, retries, and business idempotency remain application-owned. Typed unit graphs, metadata, parser/serializer buffers, and duplicate-name tracking still consume memory; collection validation occurs after allocation. No universal bounded-memory or exactly-once guarantee is implied.
Validation
Full CI passed on
27432f88: Core, Samples (.NET 8/9/10), Samples E2E, Functions runtime, Template validation, Coverage, and the final required-success gate.The latest follow-up (
d31085bd) splits Core CI into one unit-test job and isolated .NET 8/9/10 infrastructure jobs, retaining sequential infrastructure execution, CodeGen checks, merged coverage, and the required-success gate. It also restores standard NuGet root-guide packaging: README/AGENTS remain packaged, while namespace/migration guides remain repository-linked and available through the template AI bundle. CI for this follow-up is pending.Additional local verification:
Native runtime proofs use real local infrastructure and simulated BlobCreated webhook delivery because Azurite does not emit Event Grid events; Azure provisioning, authentication, and cloud delivery are not established by these tests. Dependency audit failures were resolved with patched Aspire hosting/AppHost and WebUtilities versions, without suppressing vulnerability checks.
Release-readiness checklist
XML remains deferred to Phase 5 and is not implemented in this PR.
References
docs/design/io-multiformat/coreex-io-multiformat-plan.md.src/CoreEx.IO/README.md.tests/CoreEx.Azure.Functions.TestApp/InventoryImports/README.md.