diff --git a/README.md b/README.md index 1001cb51..2604a67b 100644 --- a/README.md +++ b/README.md @@ -4,93 +4,36 @@ Database Abstraction Wrapper for Graph Schemas ![A Corgi Treat](logo_small.png) -## Purpose +DAWGS provides tools and query helpers for running property graphs on vanilla PostgreSQL without extra database +plugins. It exposes a backend abstraction for graph queries, with current backend support for PostgreSQL and Neo4j. +The query interface is built around openCypher, including a PostgreSQL SQL translator for environments that do not +support Cypher natively. -DAWGS is a collection of tools and query language helpers to enable running property graphs on vanilla PostgreSQL -without the need for additional plugins. +## Quick Start -At the core of the library is an abstraction layer that allows users to swap out existing database backends (currently -Neo4j and PostgreSQL) or build their own with no change to query implementation. The query interface is built around -openCypher with translation implementations for backends that do not natively support the query language. - -## Development Setup - -For users making changes to `dawgs` and its packages, the [go mod replace](https://go.dev/ref/mod#go-mod-file-replace) -directive can be utilized. This allows changes made in the checked out `dawgs` repo to be immediately visible to -consuming projects. - -**Example** - -``` -replace github.com/specterops/dawgs => /home/zinic/work/dawgs -``` - -### Building and Testing - -The [Makefile](Makefile) drives build and test automation. The default `make` target should suffice for normal -development processes. +Build the repository: ```bash -make +make build ``` -For validation before handing off a change, run the full test target: - -```bash -make test_all -``` - -`make test_all` runs unit tests and the integration suites. Integration tests use the `CONNECTION_STRING` environment -variable and run against the backend selected by that connection string's scheme. - -The shared integration cases under `integration/testdata/cases` and `integration/testdata/templates` are expected to be -semantically equivalent across supported backends. Avoid driver-specific skips or expected results in those files; add -driver-scoped integration coverage instead when a backend-only capability needs to be tested. - -Benign local examples: - -```bash -export CONNECTION_STRING="postgresql://dawgs:weneedbetterpasswords@localhost:65432/dawgs" -export CONNECTION_STRING="neo4j://neo4j:weneedbetterpasswords@localhost:7687" -``` - -Neo4j connection strings may use `neo4j://`, `neo4j+s://`, or `neo4j+ssc://`; a single path segment selects the Neo4j database name. - -Use `make test` for unit tests only and `make test_integration` for integration tests only. - -### Test Metrics - -`make test` writes unit test coverage artifacts under `.coverage/`: +Run unit tests: ```bash make test ``` -The stable coverage profile is `.coverage/unit.out`, and the function coverage summary is `.coverage/coverage.txt`. - -Cyclomatic complexity, CRAP, and quality signal reports are available through dedicated metric targets: +Run integration tests when a backend is available: ```bash -make complexity -make crap -make quality -make metrics +export CONNECTION_STRING="postgresql://dawgs:weneedbetterpasswords@localhost:65432/dawgs" +make test_integration ``` -`make complexity` writes `.coverage/cyclomatic.txt`. `make crap` reruns unit tests for a fresh coverage profile, then -writes `.coverage/crap.txt`, `.coverage/crap.json`, `.coverage/quality.txt`, `.coverage/quality.json`, and a standalone -HTML report at `.coverage/metrics.html`. The quality section summarizes semantic drift, backend equivalence, -integration/template invariants, fuzz health, mutation score, and benchmark drift. Signals that need external captures are -reported as pending unless their input files are provided. -Generated parser files, tests, vendor code, and testdata are excluded from these reports. The HTML report embeds its CSS -and JavaScript directly in the document, so it can be opened without network access. - -Optional quality inputs can be supplied through Make variables: +Use this module from another Go project: ```bash -make quality BACKEND_RESULT_ARGS="-backend-result pg=.coverage/integration-pg.json -backend-result neo4j=.coverage/integration-neo4j.json" -make quality BENCHMARK_REPORT=.coverage/benchmark.json BENCHMARK_BASELINE=.coverage/benchmark-baseline.json -make quality FUZZ_REPORT=.coverage/fuzz.json MUTATION_REPORT=.coverage/mutation.json +go get github.com/specterops/dawgs ``` `make quality_backend` captures PostgreSQL and Neo4j integration results for backend equivalence comparison. It requires @@ -158,20 +101,25 @@ PostgreSQL property index regression coverage is hard-failing under the `manual_ test translates Cypher to PgSQL, disables sequential scans for the `EXPLAIN`, and requires explicit node property indexes to appear in the JSON plan: -```bash -CONNECTION_STRING="postgresql://dawgs:weneedbetterpasswords@localhost:65432/dawgs" \ - go test -tags manual_integration ./integration -run TestPostgreSQLPropertyIndexPlans +For local development against a checkout, use a Go module replacement in the consuming project: + +```go +replace github.com/specterops/dawgs => /path/to/dawgs ``` -Substring and suffix predicates are intentionally not promoted to blanket schema indexes. PostgreSQL deployments can -request explicit `TextSearchIndex`/trigram property indexes for fields that need `CONTAINS`, `STARTS WITH`, or -`ENDS WITH`. The hard regression only asserts current index-compatible literal forms; dynamic parameter/property forms -that lower to helper functions are intentionally outside that contract until their lowering changes. +## Documentation -Thresholds are report-only by default. To enforce the configured thresholds, run: +- [Development workflow](docs/development.md): build, test, integration, metrics, quality, and corpus-capture commands. +- [Cypher library](cypher/README.md): parser generation and Cypher package overview. +- [PostgreSQL translation](docs/postgresql_translation.md): PostgreSQL translator behavior, optimizer lowerings, indexing notes, and validation expectations. +- [Plan corpus capture](cmd/plancorpus/README.md): shared integration corpus plan diagnostics. +- [Graph benchmark capture](cmd/graphbench/README.md): runtime diagnostics for scale scenarios. +- [Cypher syntax support](cypher/Cypher%20Syntax%20Support.md): supported Cypher behavior and semantic notes. -```bash -make metrics_check -``` +## Repository Map -The defaults can be adjusted with `CYCLO_TOP`, `CYCLO_OVER`, `CRAP_TOP`, `CRAP_OVER`, and `BENCHMARK_REGRESSION`. +- `cypher/`: parser, Cypher AST, walkers, and backend translation models. +- `drivers/`: database driver implementations. +- `integration/`: backend-equivalent integration suites and fixtures. +- `cmd/`: command-line tools for capture, export, and diagnostics. +- `tools/`: developer tools such as `dawgrun` and metrics reporting. diff --git a/cmd/plancorpus/README.md b/cmd/plancorpus/README.md index 3e49de85..75d8ee64 100644 --- a/cmd/plancorpus/README.md +++ b/cmd/plancorpus/README.md @@ -4,6 +4,11 @@ It reads `integration/testdata/cases` and `integration/testdata/templates`, loads the same datasets and inline fixtures used by the integration tests, and writes backend-specific JSONL plan records plus markdown and JSON summaries. +Use this command to baseline PostgreSQL translator and optimizer changes. PostgreSQL captures include translated SQL, +`EXPLAIN` output, plan operator counts, estimated plan cost, recursive CTE indicators, path materialization indicators, +planned lowerings, applied lowerings, skipped lowerings, and skipped-lowering reasons. Neo4j captures include logical +plan operator trees for cross-backend plan-shape comparison. + ## Usage ```bash @@ -24,3 +29,15 @@ Useful flags: | `-summary` | `.coverage/plan-corpus-summary.md` | Markdown summary | | `-summary-json` | `.coverage/plan-corpus-summary.json` | JSON summary | | `-top` | `25` | Number of expensive PostgreSQL plans to include in summaries | + +## Reviewing Captures + +The markdown summary is intended for human review. It ranks the highest-cost PostgreSQL plans, reports feature counts +such as `Recursive Union`, `SubPlan`, and `Function Scan on unnest`, and summarizes planned/applied/skipped lowerings. + +The JSON summary is intended for automation and baseline comparison. For optimizer work, check that intentional SQL +shape changes are explained and that skipped-lowering accounting remains actionable. A planned lowering without a +matching applied lowering should either have a specific skipped reason or indicate a translator consumption bug. + +Expected capture errors should be limited to invalid-query cases surfaced by the integration corpus or backend-specific +syntax differences. Unexpected capture errors should be treated as validation failures for planner or translator work. diff --git a/cypher/models/pgsql/README.md b/cypher/models/pgsql/README.md index c1b0ddf4..893a2009 100644 --- a/cypher/models/pgsql/README.md +++ b/cypher/models/pgsql/README.md @@ -1,8 +1,9 @@ # BloodHound PgSQL Model This package contains a syntax model for the PostgreSQL SQL dialect. It also contains a translation implementation to -take openCpyher input and output valid PostgreSQL SQL. This model is not intended to be a complete implementation of all -available SQL dialect features but rather the subset of the dialect required to perform openCypher to pgsql translation. +take openCypher input and output valid PostgreSQL SQL. This model is not intended to be a complete implementation of all +available SQL dialect features but rather the subset of the dialect required to perform openCypher to PostgreSQL +translation. **Expected PostgreSQL SQL dialect version**: `16.X` @@ -14,6 +15,12 @@ The `format` package contains the string rendering logic for the PgSQL syntax mo The `translate` package contains the openCypher translation implementation. +## Optimization + +The `optimize` package analyzes Cypher query shape before PostgreSQL SQL emission. See +[PostgreSQL Translation](../../../docs/postgresql_translation.md) for optimizer coverage, indexing notes, and +validation workflow. + ## Visualization The `visualization` package contains a PUML digraph formatter for the PgSQL syntax model. diff --git a/cypher/models/pgsql/optimize/OPTIMIZATION_PLAN.md b/cypher/models/pgsql/optimize/OPTIMIZATION_PLAN.md deleted file mode 100644 index 96bd21a6..00000000 --- a/cypher/models/pgsql/optimize/OPTIMIZATION_PLAN.md +++ /dev/null @@ -1,132 +0,0 @@ -# Cypher to PostgreSQL Optimization Plan - -This plan tracks optimization and rewrite work identified by running the shared integration corpus against Neo4j and PostgreSQL and comparing plan shapes. - -## Phase 1: Baseline And Tooling - -Status: completed - -- Keep a reproducible plan-capture workflow. - - Capture PostgreSQL translated SQL, PostgreSQL `EXPLAIN`, Neo4j logical plan operator trees, and optimizer planned/applied lowerings. - - Read `integration/testdata/cases` and `integration/testdata/templates`. - - Write comparable JSONL output without changing product behavior. -- Add plan-summary reporting. - - Rank cases by PostgreSQL estimated cost. - - Count plan operators, recursive CTEs, subplans, path materialization indicators, and optimizer lowerings. - - Produce markdown and JSON summaries. - -## Phase 2: Quick Wins - -Status: completed - -- Add count-store fast paths for simple count queries: - - `MATCH (n) RETURN count(n)` - - `MATCH ()-[r]->() RETURN count(r)` - - `MATCH (...) RETURN count(*)` for the same exact node and directed-edge shapes. - - Typed variants where kind filters map cleanly. - - Implemented as `CountStoreFastPath` lowering for exact node and directed-edge count shapes. -- Audit the planned/applied `PredicatePlacement` gap. - - Distinguish missing translator consumption from intentional skipped placements. - - Add explicit skipped-placement reasons when a planned lowering is not applied. - - Plan-corpus summaries now report skipped lowerings and skipped-lowering reasons. - -## Phase 3: Path Materialization - -Status: completed - -- Share path materialization for repeated path functions. - - Target `nodes(p)`, `relationships(p)`, `size(relationships(p))`, `startNode`, `endNode`, and `type`. - - Avoid repeated `SubPlan` and `Function Scan on unnest` work per path binding. - - Materialize unprojected paths once through a lateral stage when final projections return a path and its components, or repeat node-bearing component expressions. -- Expand late path materialization coverage. - - Ensure paths are built only when needed for projection, filtering, or mutation semantics. - -## Phase 4: Traversal And Recursive CTEs - -Status: completed - -- Push predicates into recursive traversal anchors and steps where semantics allow. - - Endpoint kind/property predicates. - - Relationship type predicates. - - Bound-node filters. -- Improve traversal direction selection using endpoint selectivity. - - Bound IDs. - - Labels/kinds. - - Equality predicates. - - Finite relationship type sets. - - Plan direction flips for right-endpoint binding predicates from `WHERE`, not only inline node constraints. -- Broaden limit pushdown for variable-length path queries when ordering and distinct semantics permit early termination. - -## Phase 5: Suffix And Shared Endpoint Rewrites - -Status: completed - -- Improve expansion suffix pushdown for fixed suffixes after variable-length traversals. - - Include fixed suffix steps that terminate at already-bound endpoints with inline node constraints. - - Preserve bound-endpoint constraints in the pushed terminal satisfaction check when present. -- Improve `ExpandInto` and shared endpoint rewrites for ADCS-style fanout patterns. - - Constrain earlier using bound endpoint semi-joins or correlated expansion lowering where valid. - -## Phase 6: Validation - -Status: completed - -- Add focused regression tests per optimization. - - Optimizer/lowering selection tests. - - SQL shape translation tests. - - Backend-equivalent integration tests. - - Template corpus setup now clears stale graph data before rollback-only fixture cases, keeping repeated PostgreSQL and Neo4j validation runs deterministic. -- Benchmark after each workstream. - - Run unit tests. - - Run backend-specific integration tests. - - Run plan capture and compare summary deltas. - - `quality_backend` passes against local PostgreSQL and Neo4j instances configured via environment-provided connection strings, without hardcoded credentials in docs. - - Plan corpus capture records 396 PostgreSQL plans and 396 Neo4j plans; remaining capture errors are expected invalid-query cases surfaced by both systems or Neo4j-specific parameter-map syntax rejection. - -## Phase 7: Predicate Placement Accounting - -Status: completed - -- Record planned binding-scope predicate placements when traversal constraint consumption actually pushes the matching predicate into a fixed traversal step, expansion seed, expansion edge, or expansion terminal constraint. -- Keep skipped-lowering reports focused on predicates that were not consumed by the emitted translation shape, instead of marking already-pushed traversal predicates as skipped. -- Add SQL-shape regression tests for fixed traversal and expansion-root predicate consumption. -- Refreshed plan-corpus capture applies `PredicatePlacement` in 56 of 71 planned PostgreSQL cases, reducing skipped predicate placements from 65 to 15. - -## Phase 8: Cross-Clause Predicate Placement Planning - -Status: completed - -- Stop planning traversal predicate placements for binding predicates owned by a different `MATCH` clause. -- Preserve same-clause binding predicate placement for traversal and suffix pushdown decisions. -- Refreshed plan-corpus capture now plans and applies `PredicatePlacement` in the same 56 PostgreSQL cases, removing all skipped predicate-placement reports. - -## Phase 9: Live Dataset Assumption Checks - -Status: completed - -- Re-vet optimizer assumptions against a large live PostgreSQL graph with `EXPLAIN ANALYZE`. -- Exact string property anchors now lower to `jsonb_typeof(properties -> key) = 'string'` plus `properties ->> key = value`, - allowing existing `->>` expression indexes on selective fields such as `objectid` and `name` to be used without - matching JSON booleans or numbers. -- Relationship count fast paths remain endpoint-preserving for correctness, but the PostgreSQL schema now includes a - `kind_id`-first covering edge index so typed relationship counts have a direct access path instead of relying on - endpoint-oriented traversal indexes. -- Added PG-scoped manual integration coverage for strict string equality and a read-only live-plan check that asserts - indexed `objectid` lookups use a PostgreSQL index when the connected database exposes the expected expression index. - -## Phase 10: Common Search Follow-Up - -Status: completed - -- Lower typed pattern predicates into correlated relationship `EXISTS` checks when relationship type constraints and - both endpoint correlations are sufficient, avoiding fallback CTEs for common typed existence predicates. -- Lower membership-only `collect(entity)` projections to ID arrays and rewrite membership predicates to `id = any(...)`, - keeping full entity arrays only when the collected value is otherwise observed. -- Flip single-step bound-left variable expansions toward constrained terminal kinds when there is no path binding or - continuation step, and preserve the previous-frame endpoint correlation after the flip. -- Plan shortest-path terminal-filter materialization for kind-only terminal endpoints while keeping endpoint-pair - filters limited to property/search predicates that define the pair universe. -- Defer adding blanket suffix/reverse expression indexes to schema assertion. Live common searches use `objectid` - suffix predicates, but the translator still has multiple suffix-preserving forms (`LIKE`, `cypher_ends_with`, and - null-coalesced variants). Explicit `TextSearchIndex`/trigram indexes remain available for deployments that need - substring acceleration before those semantics are unified. diff --git a/cypher/models/pgsql/optimize/lowering.go b/cypher/models/pgsql/optimize/lowering.go index f4bb36a6..20b3b2dc 100644 --- a/cypher/models/pgsql/optimize/lowering.go +++ b/cypher/models/pgsql/optimize/lowering.go @@ -6,18 +6,20 @@ import ( ) const ( - LoweringProjectionPruning = "ProjectionPruning" - LoweringLatePathMaterialization = "LatePathMaterialization" - LoweringExpandIntoDetection = "ExpandIntoDetection" - LoweringTraversalDirection = "TraversalDirectionSelection" - LoweringShortestPathStrategy = "ShortestPathStrategySelection" - LoweringShortestPathFilter = "ShortestPathFilterMaterialization" - LoweringLimitPushdown = "LimitPushdown" - LoweringExpansionSuffixPushdown = "ExpansionSuffixPushdown" - LoweringPredicatePlacement = "PredicatePlacement" - LoweringCountStoreFastPath = "CountStoreFastPath" - LoweringCollectIDMembership = "CollectIDMembership" - LoweringAggregateTraversalCount = "AggregateTraversalCount" + LoweringProjectionPruning = "ProjectionPruning" + LoweringLatePathMaterialization = "LatePathMaterialization" + LoweringExpandIntoDetection = "ExpandIntoDetection" + LoweringTraversalDirection = "TraversalDirectionSelection" + LoweringShortestPathStrategy = "ShortestPathStrategySelection" + LoweringShortestPathFilter = "ShortestPathFilterMaterialization" + LoweringLimitPushdown = "LimitPushdown" + LoweringExpansionSuffixPushdown = "ExpansionSuffixPushdown" + LoweringPredicatePlacement = "PredicatePlacement" + LoweringCountStoreFastPath = "CountStoreFastPath" + LoweringCollectIDMembership = "CollectIDMembership" + LoweringAggregateTraversalCount = "AggregateTraversalCount" + LoweringExactRangeExpansion = "ExactRangeExpansion" + LoweringPathRelationshipPredicate = "PathRelationshipPredicate" ) type LoweringDecision struct { @@ -52,6 +54,11 @@ type TraversalStepTarget struct { StepIndex int `json:"step_index"` } +type QuantifierTarget struct { + QueryPartIndex int `json:"query_part_index"` + QuantifierIndex int `json:"quantifier_index"` +} + type ProjectionPruningDecision struct { Target TraversalStepTarget `json:"target"` ReferencedSymbols []string `json:"referenced_symbols,omitempty"` @@ -164,6 +171,17 @@ type CountStoreFastPathDecision struct { KindSymbols []string `json:"kind_symbols,omitempty"` } +type ExactRangeExpansionDecision struct { + Target TraversalStepTarget `json:"target"` + Depth int64 `json:"depth"` +} + +type PathRelationshipPredicateDecision struct { + Target QuantifierTarget `json:"target"` + PathSymbol string `json:"path_symbol"` + BindingSymbol string `json:"binding_symbol"` +} + type AggregateTraversalCountDecision struct { QueryPartIndex int `json:"query_part_index"` SourceSymbol string `json:"source_symbol"` @@ -194,18 +212,20 @@ type AggregateTraversalCountShape struct { } type LoweringPlan struct { - ProjectionPruning []ProjectionPruningDecision `json:"projection_pruning,omitempty"` - LatePathMaterialization []LatePathMaterializationDecision `json:"late_path_materialization,omitempty"` - ExpandInto []ExpandIntoDecision `json:"expand_into,omitempty"` - TraversalDirection []TraversalDirectionDecision `json:"traversal_direction,omitempty"` - ShortestPathStrategy []ShortestPathStrategyDecision `json:"shortest_path_strategy,omitempty"` - ShortestPathFilter []ShortestPathFilterDecision `json:"shortest_path_filter,omitempty"` - LimitPushdown []LimitPushdownDecision `json:"limit_pushdown,omitempty"` - ExpansionSuffixPushdown []ExpansionSuffixPushdownDecision `json:"expansion_suffix_pushdown,omitempty"` - PredicatePlacement []PredicatePlacementDecision `json:"predicate_placement,omitempty"` - PatternPredicate []PatternPredicatePlacementDecision `json:"pattern_predicate_placement,omitempty"` - CountStoreFastPath []CountStoreFastPathDecision `json:"count_store_fast_path,omitempty"` - AggregateTraversalCount []AggregateTraversalCountDecision `json:"aggregate_traversal_count,omitempty"` + ProjectionPruning []ProjectionPruningDecision `json:"projection_pruning,omitempty"` + LatePathMaterialization []LatePathMaterializationDecision `json:"late_path_materialization,omitempty"` + ExpandInto []ExpandIntoDecision `json:"expand_into,omitempty"` + TraversalDirection []TraversalDirectionDecision `json:"traversal_direction,omitempty"` + ShortestPathStrategy []ShortestPathStrategyDecision `json:"shortest_path_strategy,omitempty"` + ShortestPathFilter []ShortestPathFilterDecision `json:"shortest_path_filter,omitempty"` + LimitPushdown []LimitPushdownDecision `json:"limit_pushdown,omitempty"` + ExpansionSuffixPushdown []ExpansionSuffixPushdownDecision `json:"expansion_suffix_pushdown,omitempty"` + PredicatePlacement []PredicatePlacementDecision `json:"predicate_placement,omitempty"` + PatternPredicate []PatternPredicatePlacementDecision `json:"pattern_predicate_placement,omitempty"` + CountStoreFastPath []CountStoreFastPathDecision `json:"count_store_fast_path,omitempty"` + ExactRangeExpansion []ExactRangeExpansionDecision `json:"exact_range_expansion,omitempty"` + PathRelationshipPredicate []PathRelationshipPredicateDecision `json:"path_relationship_predicate,omitempty"` + AggregateTraversalCount []AggregateTraversalCountDecision `json:"aggregate_traversal_count,omitempty"` } func (s LoweringPlan) Empty() bool { @@ -220,6 +240,8 @@ func (s LoweringPlan) Empty() bool { len(s.PredicatePlacement) == 0 && len(s.PatternPredicate) == 0 && len(s.CountStoreFastPath) == 0 && + len(s.ExactRangeExpansion) == 0 && + len(s.PathRelationshipPredicate) == 0 && len(s.AggregateTraversalCount) == 0 } @@ -241,6 +263,8 @@ func (s LoweringPlan) Decisions() []LoweringDecision { add(LoweringExpansionSuffixPushdown, len(s.ExpansionSuffixPushdown) > 0) add(LoweringPredicatePlacement, len(s.PredicatePlacement) > 0 || len(s.PatternPredicate) > 0) add(LoweringCountStoreFastPath, len(s.CountStoreFastPath) > 0) + add(LoweringExactRangeExpansion, len(s.ExactRangeExpansion) > 0) + add(LoweringPathRelationshipPredicate, len(s.PathRelationshipPredicate) > 0) add(LoweringAggregateTraversalCount, len(s.AggregateTraversalCount) > 0) return decisions diff --git a/cypher/models/pgsql/optimize/lowering_plan.go b/cypher/models/pgsql/optimize/lowering_plan.go index d1cf5122..61df8831 100644 --- a/cypher/models/pgsql/optimize/lowering_plan.go +++ b/cypher/models/pgsql/optimize/lowering_plan.go @@ -4,6 +4,7 @@ import ( "strings" "github.com/specterops/dawgs/cypher/models/cypher" + "github.com/specterops/dawgs/cypher/models/walk" "github.com/specterops/dawgs/graph" ) @@ -38,6 +39,8 @@ const ( boundSourceSelectivityTopN ) +const maxExactRangeExpansionDepth int64 = 2 + func BuildLoweringPlan(query *cypher.RegularQuery, predicateAttachments []PredicateAttachment) (LoweringPlan, error) { if query == nil || query.SingleQuery == nil { return LoweringPlan{}, nil @@ -107,6 +110,9 @@ func appendQueryPartLowerings( return err } + appendExactRangeExpansionDecisions(plan, queryPartIndex, readingClauses) + appendPatternPredicateExactRangeExpansionDecisions(plan, queryPartIndex, queryPart) + appendPathRelationshipPredicateDecisions(plan, queryPartIndex, queryPart) appendProjectionPruningDecisions(plan, queryPartIndex, readingClauses, sourceReferences) appendLatePathMaterializationDecisions(plan, queryPartIndex, readingClauses, sourceReferences) appendPatternPredicateProjectionLowerings(plan, queryPartIndex, queryPart, sourceReferences) @@ -121,6 +127,219 @@ func appendQueryPartLowerings( return nil } +func appendExactRangeExpansionDecisions(plan *LoweringPlan, queryPartIndex int, readingClauses []*cypher.ReadingClause) { + for clauseIndex, readingClause := range readingClauses { + if readingClause == nil || readingClause.Match == nil || readingClause.Match.Optional { + continue + } + + for patternIndex, patternPart := range readingClause.Match.Pattern { + appendPatternExactRangeExpansionDecisions(plan, PatternTarget{ + QueryPartIndex: queryPartIndex, + ClauseIndex: clauseIndex, + PatternIndex: patternIndex, + }, patternPart) + } + } +} + +func appendPatternPredicateExactRangeExpansionDecisions(plan *LoweringPlan, queryPartIndex int, queryPart cypher.SyntaxNode) { + for _, indexedPredicate := range indexedPatternPredicatesInQueryPart(queryPart) { + patternPart := patternPartForPredicate(indexedPredicate.Predicate) + appendPatternExactRangeExpansionDecisions(plan, PatternTarget{ + QueryPartIndex: queryPartIndex, + ClauseIndex: indexedPredicate.ClauseIndex, + PatternIndex: indexedPredicate.PredicateIndex, + Predicate: true, + PredicateIndex: indexedPredicate.PredicateIndex, + }, patternPart) + } +} + +func appendPatternExactRangeExpansionDecisions(plan *LoweringPlan, target PatternTarget, patternPart *cypher.PatternPart) { + for stepIndex, step := range traversalStepsForPattern(patternPart) { + if exactRangeExpansionCandidate(patternPart, step) { + plan.ExactRangeExpansion = append(plan.ExactRangeExpansion, ExactRangeExpansionDecision{ + Target: target.TraversalStep(stepIndex), + Depth: ExactPatternRangeDepth(step.Relationship.Range), + }) + } + } +} + +func exactRangeExpansionCandidate(patternPart *cypher.PatternPart, step sourceTraversalStep) bool { + if patternPart == nil { + return false + } + + if patternPart.ShortestPathPattern || + patternPart.AllShortestPathsPattern || + step.Relationship == nil || + step.Relationship.Direction == graph.DirectionBoth || + step.Relationship.Variable != nil { + return false + } + + depth := ExactPatternRangeDepth(step.Relationship.Range) + return depth >= 1 && depth <= maxExactRangeExpansionDepth +} + +func hasExactRangeExpansionDecision(plan *LoweringPlan, target TraversalStepTarget) bool { + if plan == nil { + return false + } + + for _, decision := range plan.ExactRangeExpansion { + if decision.Target == target { + return true + } + } + + return false +} + +func ExactPatternRangeDepth(patternRange *cypher.PatternRange) int64 { + if patternRange == nil || patternRange.StartIndex == nil || patternRange.EndIndex == nil { + return 0 + } + + if *patternRange.StartIndex != *patternRange.EndIndex { + return 0 + } + + return *patternRange.StartIndex +} + +type indexedQuantifier struct { + Index int + Quantifier *cypher.Quantifier +} + +type quantifierCollector struct { + walk.VisitorHandler + quantifiers []indexedQuantifier +} + +func (s *quantifierCollector) Enter(node cypher.SyntaxNode) { + if quantifier, isQuantifier := node.(*cypher.Quantifier); isQuantifier { + s.quantifiers = append(s.quantifiers, indexedQuantifier{ + Index: len(s.quantifiers), + Quantifier: quantifier, + }) + } +} + +func (s *quantifierCollector) Visit(cypher.SyntaxNode) {} +func (s *quantifierCollector) Exit(cypher.SyntaxNode) {} + +func indexedQuantifiersInQueryPart(queryPart cypher.SyntaxNode) []indexedQuantifier { + if queryPart == nil { + return nil + } + + collector := &quantifierCollector{ + VisitorHandler: walk.NewCancelableErrorHandler(), + } + + if err := walk.Cypher(queryPart, collector); err != nil { + return nil + } + + return collector.quantifiers +} + +func quantifiersInSyntax(node cypher.SyntaxNode) []*cypher.Quantifier { + if node == nil { + return nil + } + + var ( + quantifiers []*cypher.Quantifier + collector = &quantifierCollector{ + VisitorHandler: walk.NewCancelableErrorHandler(), + } + ) + + if err := walk.Cypher(node, collector); err != nil { + return nil + } + + for _, indexed := range collector.quantifiers { + quantifiers = append(quantifiers, indexed.Quantifier) + } + + return quantifiers +} + +func pathRelationshipQuantifierCandidate(quantifier *cypher.Quantifier) (string, string, bool) { + if quantifier == nil || + (quantifier.Type != cypher.QuantifierTypeAny && quantifier.Type != cypher.QuantifierTypeNone) || + quantifier.Filter == nil || + quantifier.Filter.Specifier == nil || + quantifier.Filter.Specifier.Variable == nil { + return "", "", false + } + + function, isFunction := quantifier.Filter.Specifier.Expression.(*cypher.FunctionInvocation) + if !isFunction || function == nil || function.NumArguments() != 1 || !strings.EqualFold(function.Name, cypher.RelationshipsFunction) { + return "", "", false + } + + pathVariable, isPathVariable := function.Arguments[0].(*cypher.Variable) + if !isPathVariable || pathVariable == nil || pathVariable.Symbol == "" { + return "", "", false + } + + bindingSymbol := quantifier.Filter.Specifier.Variable.Symbol + if bindingSymbol == "" { + return "", "", false + } + + return pathVariable.Symbol, bindingSymbol, true +} + +func appendPathRelationshipPredicateDecisions(plan *LoweringPlan, queryPartIndex int, queryPart cypher.SyntaxNode) { + quantifierIndexes := map[*cypher.Quantifier]int{} + for _, indexed := range indexedQuantifiersInQueryPart(queryPart) { + quantifierIndexes[indexed.Quantifier] = indexed.Index + } + + appendMatchWhereQuantifier := func(quantifier *cypher.Quantifier) { + if quantifierIndex, indexed := quantifierIndexes[quantifier]; indexed { + if pathSymbol, bindingSymbol, eligible := pathRelationshipQuantifierCandidate(quantifier); eligible { + plan.PathRelationshipPredicate = append(plan.PathRelationshipPredicate, PathRelationshipPredicateDecision{ + Target: QuantifierTarget{ + QueryPartIndex: queryPartIndex, + QuantifierIndex: quantifierIndex, + }, + PathSymbol: pathSymbol, + BindingSymbol: bindingSymbol, + }) + } + } + } + + appendReadingClauseDecisions := func(readingClauses []*cypher.ReadingClause) { + for _, readingClause := range readingClauses { + if readingClause == nil || readingClause.Match == nil || readingClause.Match.Where == nil { + continue + } + + for _, quantifier := range quantifiersInSyntax(readingClause.Match.Where) { + appendMatchWhereQuantifier(quantifier) + } + } + } + + switch typedQueryPart := queryPart.(type) { + case *cypher.SinglePartQuery: + appendReadingClauseDecisions(typedQueryPart.ReadingClauses) + + case *cypher.MultiPartQueryPart: + appendReadingClauseDecisions(typedQueryPart.ReadingClauses) + } +} + func appendProjectionPruningDecisions(plan *LoweringPlan, queryPartIndex int, readingClauses []*cypher.ReadingClause, sourceReferences map[string]struct{}) { for clauseIndex, readingClause := range readingClauses { if readingClause == nil || readingClause.Match == nil || readingClause.Match.Optional { @@ -156,7 +375,7 @@ func appendPatternProjectionPruningDecisions(plan *LoweringPlan, target PatternT hasPruning bool ) - if step.Relationship.Range != nil { + if step.Relationship.Range != nil && !hasExactRangeExpansionDecision(plan, decision.Target) { decision.OmitRelationship = !edgeReferenced decision.OmitPathBinding = !pathReferenced hasPruning = decision.OmitRelationship || decision.OmitPathBinding @@ -266,7 +485,7 @@ func appendPatternLatePathMaterializationDecisions(plan *LoweringPlan, target Pa for stepIndex, step := range steps { stepTarget := target.TraversalStep(stepIndex) - if step.Relationship.Range != nil { + if step.Relationship.Range != nil && !hasExactRangeExpansionDecision(plan, stepTarget) { if !pathReferenced { continue } @@ -1277,6 +1496,10 @@ func appendExpansionSuffixPushdownDecisions(plan *LoweringPlan, queryPartIndex i ClauseIndex: clauseIndex, PatternIndex: patternIndex, }.TraversalStep(stepIndex) + if hasExactRangeExpansionDecision(plan, target) { + continue + } + if hasTraversalDirectionFlip(plan, target) || expansionStepMayFlipForConstraintBalance(stepIndex, step, declaredEndpoints[stepIndex]) { continue } diff --git a/cypher/models/pgsql/optimize/optimizer_test.go b/cypher/models/pgsql/optimize/optimizer_test.go index 11b7399b..33848399 100644 --- a/cypher/models/pgsql/optimize/optimizer_test.go +++ b/cypher/models/pgsql/optimize/optimizer_test.go @@ -425,6 +425,300 @@ func TestLoweringPlanReportsLatePathMaterialization(t *testing.T) { }) } +func TestLoweringPlanReportsExactOneHopRangeExpansion(t *testing.T) { + t.Parallel() + + regularQuery, err := frontend.ParseCypher(frontend.NewContext(), ` + MATCH p = (n)-[:MemberOf*1..1]->(m) + RETURN p + `) + require.NoError(t, err) + + plan, err := Optimize(regularQuery) + require.NoError(t, err) + require.Contains(t, plan.LoweringPlan.Decisions(), LoweringDecision{ + Name: LoweringExactRangeExpansion, + }) + require.Equal(t, []ExactRangeExpansionDecision{{ + Target: TraversalStepTarget{ + QueryPartIndex: 0, + ClauseIndex: 0, + PatternIndex: 0, + StepIndex: 0, + }, + Depth: 1, + }}, plan.LoweringPlan.ExactRangeExpansion) + require.Contains(t, plan.LoweringPlan.LatePathMaterialization, LatePathMaterializationDecision{ + Target: TraversalStepTarget{ + QueryPartIndex: 0, + ClauseIndex: 0, + PatternIndex: 0, + StepIndex: 0, + }, + Mode: LatePathMaterializationPathEdgeID, + }) +} + +func TestLoweringPlanReportsExactTwoHopRangeExpansion(t *testing.T) { + t.Parallel() + + regularQuery, err := frontend.ParseCypher(frontend.NewContext(), ` + MATCH p = (n)-[:MemberOf*2..2]->(m) + RETURN p + `) + require.NoError(t, err) + + plan, err := Optimize(regularQuery) + require.NoError(t, err) + require.Contains(t, plan.LoweringPlan.Decisions(), LoweringDecision{ + Name: LoweringExactRangeExpansion, + }) + require.Equal(t, []ExactRangeExpansionDecision{{ + Target: TraversalStepTarget{ + QueryPartIndex: 0, + ClauseIndex: 0, + PatternIndex: 0, + StepIndex: 0, + }, + Depth: 2, + }}, plan.LoweringPlan.ExactRangeExpansion) +} + +func TestExactRangeDependentPlanningRequiresDecision(t *testing.T) { + t.Parallel() + + regularQuery, err := frontend.ParseCypher(frontend.NewContext(), ` + MATCH p = (n)-[:MemberOf*1..1]->(m)-[:Enroll]->(ca) + RETURN p + `) + require.NoError(t, err) + + var ( + readingClauses = regularQuery.SingleQuery.SinglePartQuery.ReadingClauses + patternPart = readingClauses[0].Match.Pattern[0] + steps = traversalStepsForPattern(patternPart) + target = PatternTarget{} + sourceReferences = map[string]struct{}{ + "p": {}, + } + ) + + t.Run("without decision", func(t *testing.T) { + plan := LoweringPlan{} + + appendPatternProjectionPruningDecisions(&plan, target, patternPart, steps, sourceReferences) + require.Equal(t, []ProjectionPruningDecision{ + { + Target: target.TraversalStep(0), + ReferencedSymbols: []string{"p"}, + PatternBindingReferenced: true, + OmitRelationship: true, + }, + }, plan.ProjectionPruning) + + appendPatternLatePathMaterializationDecisions(&plan, target, patternPart, steps, sourceReferences) + require.Contains(t, plan.LatePathMaterialization, LatePathMaterializationDecision{ + Target: target.TraversalStep(0), + Mode: LatePathMaterializationExpansionPath, + }) + + appendExpansionSuffixPushdownDecisions(&plan, 0, readingClauses) + require.Contains(t, plan.ExpansionSuffixPushdown, ExpansionSuffixPushdownDecision{ + Target: target.TraversalStep(0), + SuffixLength: 1, + SuffixStartStep: 1, + SuffixEndStep: 1, + }) + }) + + t.Run("with decision", func(t *testing.T) { + plan := LoweringPlan{ + ExactRangeExpansion: []ExactRangeExpansionDecision{ + { + Target: target.TraversalStep(0), + Depth: 1, + }, + }, + } + + appendPatternProjectionPruningDecisions(&plan, target, patternPart, steps, sourceReferences) + require.Empty(t, plan.ProjectionPruning) + + appendPatternLatePathMaterializationDecisions(&plan, target, patternPart, steps, sourceReferences) + require.Contains(t, plan.LatePathMaterialization, LatePathMaterializationDecision{ + Target: target.TraversalStep(0), + Mode: LatePathMaterializationPathEdgeID, + }) + + appendExpansionSuffixPushdownDecisions(&plan, 0, readingClauses) + require.Empty(t, plan.ExpansionSuffixPushdown) + }) +} + +func TestLoweringPlanSkipsExactRangeExpansionBeyondDepthCap(t *testing.T) { + t.Parallel() + + regularQuery, err := frontend.ParseCypher(frontend.NewContext(), ` + MATCH p = (n)-[:MemberOf*3..3]->(m) + RETURN p + `) + require.NoError(t, err) + + plan, err := Optimize(regularQuery) + require.NoError(t, err) + require.NotContains(t, plan.LoweringPlan.Decisions(), LoweringDecision{ + Name: LoweringExactRangeExpansion, + }) + require.Empty(t, plan.LoweringPlan.ExactRangeExpansion) +} + +func TestLoweringPlanSkipsUndirectedExactRangeExpansion(t *testing.T) { + t.Parallel() + + regularQuery, err := frontend.ParseCypher(frontend.NewContext(), ` + MATCH p = (n)-[:MemberOf*2..2]-(m) + RETURN p + `) + require.NoError(t, err) + + plan, err := Optimize(regularQuery) + require.NoError(t, err) + require.NotContains(t, plan.LoweringPlan.Decisions(), LoweringDecision{ + Name: LoweringExactRangeExpansion, + }) + require.Empty(t, plan.LoweringPlan.ExactRangeExpansion) +} + +func TestLoweringPlanSkipsExactOneHopRangeExpansionForNamedRelationshipBinding(t *testing.T) { + t.Parallel() + + regularQuery, err := frontend.ParseCypher(frontend.NewContext(), ` + MATCH p = (n)-[r:MemberOf*1..1]->(m) + RETURN p, r + `) + require.NoError(t, err) + + plan, err := Optimize(regularQuery) + require.NoError(t, err) + require.NotContains(t, plan.LoweringPlan.Decisions(), LoweringDecision{ + Name: LoweringExactRangeExpansion, + }) + require.Empty(t, plan.LoweringPlan.ExactRangeExpansion) + require.Contains(t, plan.LoweringPlan.LatePathMaterialization, LatePathMaterializationDecision{ + Target: TraversalStepTarget{ + QueryPartIndex: 0, + ClauseIndex: 0, + PatternIndex: 0, + StepIndex: 0, + }, + Mode: LatePathMaterializationExpansionPath, + }) +} + +func TestLoweringPlanSkipsExactOneHopRangeExpansionForShortestPath(t *testing.T) { + t.Parallel() + + regularQuery, err := frontend.ParseCypher(frontend.NewContext(), ` + MATCH p = shortestPath((n)-[:MemberOf*1..1]->(m)) + RETURN p + `) + require.NoError(t, err) + + plan, err := Optimize(regularQuery) + require.NoError(t, err) + require.NotContains(t, plan.LoweringPlan.Decisions(), LoweringDecision{ + Name: LoweringExactRangeExpansion, + }) + require.Empty(t, plan.LoweringPlan.ExactRangeExpansion) +} + +func TestLoweringPlanReportsPathRelationshipPredicate(t *testing.T) { + t.Parallel() + + regularQuery, err := frontend.ParseCypher(frontend.NewContext(), ` + MATCH p = (n)-[:MemberOf*1..]->(m) + WHERE any(r in relationships(p) WHERE type(r) STARTS WITH 'Member') + RETURN p + `) + require.NoError(t, err) + + plan, err := Optimize(regularQuery) + require.NoError(t, err) + require.Contains(t, plan.LoweringPlan.Decisions(), LoweringDecision{ + Name: LoweringPathRelationshipPredicate, + }) + require.Equal(t, []PathRelationshipPredicateDecision{{ + Target: QuantifierTarget{ + QueryPartIndex: 0, + QuantifierIndex: 0, + }, + PathSymbol: "p", + BindingSymbol: "r", + }}, plan.LoweringPlan.PathRelationshipPredicate) +} + +func TestLoweringPlanReportsNonePathRelationshipPredicate(t *testing.T) { + t.Parallel() + + regularQuery, err := frontend.ParseCypher(frontend.NewContext(), ` + MATCH p = (n)-[:MemberOf*1..]->(m) + WHERE none(r in relationships(p) WHERE type(r) = 'AdminTo') + RETURN p + `) + require.NoError(t, err) + + plan, err := Optimize(regularQuery) + require.NoError(t, err) + require.Contains(t, plan.LoweringPlan.Decisions(), LoweringDecision{ + Name: LoweringPathRelationshipPredicate, + }) + require.Equal(t, []PathRelationshipPredicateDecision{{ + Target: QuantifierTarget{ + QueryPartIndex: 0, + QuantifierIndex: 0, + }, + PathSymbol: "p", + BindingSymbol: "r", + }}, plan.LoweringPlan.PathRelationshipPredicate) +} + +func TestLoweringPlanSkipsPathRelationshipPredicateForAllQuantifier(t *testing.T) { + t.Parallel() + + regularQuery, err := frontend.ParseCypher(frontend.NewContext(), ` + MATCH p = (n)-[:MemberOf*1..]->(m) + WHERE all(r in relationships(p) WHERE type(r) = 'MemberOf') + RETURN p + `) + require.NoError(t, err) + + plan, err := Optimize(regularQuery) + require.NoError(t, err) + require.NotContains(t, plan.LoweringPlan.Decisions(), LoweringDecision{ + Name: LoweringPathRelationshipPredicate, + }) + require.Empty(t, plan.LoweringPlan.PathRelationshipPredicate) +} + +func TestLoweringPlanSkipsPathRelationshipPredicateAfterWithProjection(t *testing.T) { + t.Parallel() + + regularQuery, err := frontend.ParseCypher(frontend.NewContext(), ` + MATCH p = (n)-[:MemberOf*1..]->(m) + WITH p + WHERE none(r in relationships(p) WHERE type(r) = 'AdminTo') + RETURN p + `) + require.NoError(t, err) + + plan, err := Optimize(regularQuery) + require.NoError(t, err) + require.NotContains(t, plan.LoweringPlan.Decisions(), LoweringDecision{ + Name: LoweringPathRelationshipPredicate, + }) + require.Empty(t, plan.LoweringPlan.PathRelationshipPredicate) +} + func TestLoweringPlanReportsExpansionSuffixPushdown(t *testing.T) { t.Parallel() @@ -1514,8 +1808,14 @@ func TestConservativePatternReorderingMovesIndependentNodeAnchorsEarlier(t *test plan, err := Optimize(regularQuery) require.NoError(t, err) require.Equal(t, []RuleResult{ - {Name: "ConservativePatternReordering", Applied: true}, - {Name: "PredicateAttachment", Applied: false}, + { + Name: "ConservativePatternReordering", + Applied: true, + }, + { + Name: "PredicateAttachment", + Applied: false, + }, }, plan.Rules) readingClauses := plan.Query.SingleQuery.SinglePartQuery.ReadingClauses @@ -1538,8 +1838,73 @@ func TestConservativePatternReorderingKeepsDependentAnchorsInPlace(t *testing.T) plan, err := Optimize(regularQuery) require.NoError(t, err) require.Equal(t, []RuleResult{ - {Name: "ConservativePatternReordering", Applied: false}, - {Name: "PredicateAttachment", Applied: true}, + { + Name: "ConservativePatternReordering", + Applied: false, + }, + { + Name: "PredicateAttachment", + Applied: true, + }, + }, plan.Rules) + + readingClauses := plan.Query.SingleQuery.SinglePartQuery.ReadingClauses + require.Equal(t, "a", firstNodeSymbol(readingClauses[0])) + require.Equal(t, "b", firstNodeSymbol(readingClauses[1])) +} + +func TestConservativePatternReorderingUsesSelectivityWithinDependencySafeRegion(t *testing.T) { + t.Parallel() + + regularQuery, err := frontend.ParseCypher(frontend.NewContext(), ` + MATCH (a:Group) + MATCH (b:User {objectid: 'target'}) + MATCH p = (a)-[:MemberOf]->(b) + RETURN p + `) + require.NoError(t, err) + + plan, err := Optimize(regularQuery) + require.NoError(t, err) + require.Equal(t, []RuleResult{ + { + Name: "ConservativePatternReordering", + Applied: true, + }, + { + Name: "PredicateAttachment", + Applied: false, + }, + }, plan.Rules) + + readingClauses := plan.Query.SingleQuery.SinglePartQuery.ReadingClauses + require.Equal(t, "b", firstNodeSymbol(readingClauses[0])) + require.Equal(t, "a", firstNodeSymbol(readingClauses[1])) + require.Len(t, readingClauses[2].Match.Pattern[0].PatternElements, 3) +} + +func TestConservativePatternReorderingPinsUnresolvedExternalDependencies(t *testing.T) { + t.Parallel() + + regularQuery, err := frontend.ParseCypher(frontend.NewContext(), ` + MATCH (a) + WHERE a.name = external.name + MATCH (b:User {objectid: 'target'}) + RETURN b + `) + require.NoError(t, err) + + plan, err := Optimize(regularQuery) + require.NoError(t, err) + require.Equal(t, []RuleResult{ + { + Name: "ConservativePatternReordering", + Applied: false, + }, + { + Name: "PredicateAttachment", + Applied: true, + }, }, plan.Rules) readingClauses := plan.Query.SingleQuery.SinglePartQuery.ReadingClauses diff --git a/cypher/models/pgsql/optimize/reordering.go b/cypher/models/pgsql/optimize/reordering.go index 4a380108..83ae4bda 100644 --- a/cypher/models/pgsql/optimize/reordering.go +++ b/cypher/models/pgsql/optimize/reordering.go @@ -1,10 +1,6 @@ package optimize -import ( - "sort" - - "github.com/specterops/dawgs/cypher/models/cypher" -) +import "github.com/specterops/dawgs/cypher/models/cypher" type ConservativePatternReorderingRule struct{} @@ -29,9 +25,12 @@ func (s ConservativePatternReorderingRule) Apply(plan *Plan) (bool, error) { } type reorderCandidate struct { - clause *cypher.ReadingClause - rank int - index int + clause *cypher.ReadingClause + declarations map[string]struct{} + dependencies map[string]struct{} + movable bool + score boundSourceSelectivity + index int } func reorderMultiPartQuery(query *cypher.MultiPartQuery, analysis Analysis) bool { @@ -82,84 +81,229 @@ func reorderReadingClauses(readingClauses []*cypher.ReadingClause, regions []Reg continue } - applied = reorderRegion(readingClauses[region.StartClause:region.EndClause+1]) || applied + applied = reorderRegion(readingClauses[region.StartClause:region.EndClause+1], declaredBeforeClause(readingClauses, region.StartClause)) || applied + } + + return applied +} + +func reorderRegion(regionClauses []*cypher.ReadingClause, declaredBeforeRegion map[string]struct{}) bool { + candidates := reorderCandidates(regionClauses, declaredBeforeRegion) + + reordered := reorderCandidateSegments(candidates, declaredBeforeRegion) + + var applied bool + for idx, candidate := range reordered { + if regionClauses[idx] != candidate.clause { + applied = true + regionClauses[idx] = candidate.clause + } } return applied } -func reorderRegion(regionClauses []*cypher.ReadingClause) bool { +func declaredBeforeClause(readingClauses []*cypher.ReadingClause, clauseIndex int) map[string]struct{} { + declared := map[string]struct{}{} + for idx := 0; idx < clauseIndex && idx < len(readingClauses); idx++ { + for _, binding := range bindingsForReadingClause(idx, readingClauses[idx]) { + declared[binding.Symbol] = struct{}{} + } + } + + return declared +} + +func reorderCandidates(regionClauses []*cypher.ReadingClause, declaredBeforeRegion map[string]struct{}) []reorderCandidate { var ( - candidates = make([]reorderCandidate, len(regionClauses)) - declaredBefore = map[string]struct{}{} + candidates = make([]reorderCandidate, len(regionClauses)) + firstDeclaration = copyStringSet(declaredBeforeRegion) + regionSymbols = map[string]struct{}{} ) for idx, clause := range regionClauses { - candidates[idx] = reorderCandidate{ - clause: clause, - rank: matchClauseRank(clause, declaredBefore), - index: idx, + for _, binding := range bindingsForReadingClause(idx, clause) { + regionSymbols[binding.Symbol] = struct{}{} } + } + + for idx, clause := range regionClauses { + var ( + declarations = map[string]struct{}{} + dependencies = map[string]struct{}{} + ) for _, binding := range bindingsForReadingClause(idx, clause) { - declaredBefore[binding.Symbol] = struct{}{} + if _, declared := firstDeclaration[binding.Symbol]; declared { + dependencies[binding.Symbol] = struct{}{} + continue + } + + firstDeclaration[binding.Symbol] = struct{}{} + declarations[binding.Symbol] = struct{}{} } - } - sort.SliceStable(candidates, func(i, j int) bool { - return candidates[i].rank < candidates[j].rank - }) + var match *cypher.Match + if clause != nil { + match = clause.Match + } - var applied bool - for idx, candidate := range candidates { - if regionClauses[idx] != candidate.clause { - applied = true - regionClauses[idx] = candidate.clause + for _, dependency := range localMatchDependencies(match) { + dependencies[dependency] = struct{}{} + } + + movable := true + for dependency := range dependencies { + if _, declaredBefore := declaredBeforeRegion[dependency]; declaredBefore { + continue + } + if _, declaredInRegion := regionSymbols[dependency]; declaredInRegion { + continue + } + + movable = false + break + } + + candidates[idx] = reorderCandidate{ + clause: clause, + declarations: declarations, + dependencies: dependencies, + movable: movable, + score: matchClauseSelectivity(clause), + index: idx, } } - return applied + return candidates } -func matchClauseRank(readingClause *cypher.ReadingClause, declaredBefore map[string]struct{}) int { - if isIndependentNodeAnchor(readingClause, declaredBefore) { - return 0 +func reorderCandidateSegments(candidates []reorderCandidate, declaredBeforeRegion map[string]struct{}) []reorderCandidate { + var ( + reordered = make([]reorderCandidate, 0, len(candidates)) + available = copyStringSet(declaredBeforeRegion) + segment []reorderCandidate + ) + + flushSegment := func() { + if len(segment) == 0 { + return + } + + nextSegment := reorderMovableCandidates(segment, available) + for _, candidate := range nextSegment { + mergeStringSet(available, candidate.declarations) + } + + reordered = append(reordered, nextSegment...) + segment = nil } - return 1 + for _, candidate := range candidates { + if !candidate.movable { + flushSegment() + reordered = append(reordered, candidate) + mergeStringSet(available, candidate.declarations) + continue + } + + segment = append(segment, candidate) + } + + flushSegment() + return reordered } -func isIndependentNodeAnchor(readingClause *cypher.ReadingClause, declaredBefore map[string]struct{}) bool { - if readingClause == nil || readingClause.Match == nil { - return false +func reorderMovableCandidates(candidates []reorderCandidate, available map[string]struct{}) []reorderCandidate { + var ( + reordered = make([]reorderCandidate, 0, len(candidates)) + remaining = append([]reorderCandidate(nil), candidates...) + ) + + for len(remaining) > 0 { + nextIndex := bestSchedulableCandidateIndex(remaining, available) + if nextIndex < 0 { + reordered = append(reordered, remaining...) + break + } + + next := remaining[nextIndex] + reordered = append(reordered, next) + mergeStringSet(available, next.declarations) + remaining = append(remaining[:nextIndex], remaining[nextIndex+1:]...) } - match := readingClause.Match - if match.Optional || len(match.Pattern) != 1 { - return false + return reordered +} + +func bestSchedulableCandidateIndex(candidates []reorderCandidate, available map[string]struct{}) int { + bestIndex := -1 + for idx, candidate := range candidates { + if !candidateDependenciesSatisfied(candidate, available) { + continue + } + + if bestIndex < 0 || + candidate.score > candidates[bestIndex].score || + (candidate.score == candidates[bestIndex].score && candidate.index < candidates[bestIndex].index) { + bestIndex = idx + } } - nodePattern, ok := singleNodePattern(match.Pattern[0]) - if !ok || nodePattern.Variable == nil || nodePattern.Variable.Symbol == "" { + return bestIndex +} + +func candidateDependenciesSatisfied(candidate reorderCandidate, available map[string]struct{}) bool { + for dependency := range candidate.dependencies { + if _, declared := available[dependency]; declared { + continue + } + if _, declaredByCandidate := candidate.declarations[dependency]; declaredByCandidate { + continue + } + return false } - if _, alreadyDeclared := declaredBefore[nodePattern.Variable.Symbol]; alreadyDeclared { - return false + return true +} + +func matchClauseSelectivity(readingClause *cypher.ReadingClause) boundSourceSelectivity { + if readingClause == nil || readingClause.Match == nil { + return boundSourceSelectivityNone } - if !isSelectiveNodeAnchor(nodePattern, match.Where) { - return false + var selectivity boundSourceSelectivity + for _, patternPart := range readingClause.Match.Pattern { + for _, nodePattern := range nodePatternsForPattern(patternPart) { + mergeSelectivityValue(&selectivity, nodePatternSelectivity(nodePattern, false)) + } + + for _, step := range traversalStepsForPattern(patternPart) { + if step.Relationship == nil { + continue + } + + if len(step.Relationship.Kinds) > 0 { + mergeSelectivityValue(&selectivity, boundSourceSelectivityKindOnly) + } + mergeSelectivityValue(&selectivity, propertyConstraintSelectivity(step.Relationship.Properties)) + } } - declared := bindingSymbolSet(bindingsForMatch(0, match)) - for _, dependency := range localMatchDependencies(match) { - if _, isLocal := declared[dependency]; !isLocal { - return false + if readingClause.Match.Where != nil { + for _, expression := range readingClause.Match.Where.Expressions { + for _, term := range cypherConjunctionTerms(expression) { + if _, termSelectivity, ok := propertyPredicateSelectivity(term); ok { + mergeSelectivityValue(&selectivity, termSelectivity) + } else { + mergeSelectivityValue(&selectivity, boundSourceSelectivityPredicate) + } + } } } - return true + return selectivity } func singleNodePattern(pattern *cypher.PatternPart) (*cypher.NodePattern, bool) { @@ -170,10 +314,6 @@ func singleNodePattern(pattern *cypher.PatternPart) (*cypher.NodePattern, bool) return pattern.PatternElements[0].AsNodePattern() } -func isSelectiveNodeAnchor(nodePattern *cypher.NodePattern, where *cypher.Where) bool { - return len(nodePattern.Kinds) > 0 || nodePattern.Properties != nil || wherePredicateCount(where) > 0 -} - func localMatchDependencies(match *cypher.Match) []string { if match == nil { return nil @@ -202,15 +342,6 @@ func localMatchDependencies(match *cypher.Match) []string { return sortedUniqueStrings(dependencies) } -func bindingSymbolSet(bindings []Binding) map[string]struct{} { - symbols := make(map[string]struct{}, len(bindings)) - for _, binding := range bindings { - symbols[binding.Symbol] = struct{}{} - } - - return symbols -} - func bindingsForReadingClause(clauseIndex int, readingClause *cypher.ReadingClause) []Binding { if readingClause == nil || readingClause.Match == nil { return nil @@ -218,3 +349,9 @@ func bindingsForReadingClause(clauseIndex int, readingClause *cypher.ReadingClau return bindingsForMatch(clauseIndex, readingClause.Match) } + +func mergeStringSet(dst map[string]struct{}, src map[string]struct{}) { + for value := range src { + dst[value] = struct{}{} + } +} diff --git a/cypher/models/pgsql/test/translation_cases/pattern_binding.sql b/cypher/models/pgsql/test/translation_cases/pattern_binding.sql index 2d97d25f..c32946cb 100644 --- a/cypher/models/pgsql/test/translation_cases/pattern_binding.sql +++ b/cypher/models/pgsql/test/translation_cases/pattern_binding.sql @@ -27,7 +27,10 @@ with s0 as (select e0.id as e0, (n0.id, n0.kind_ids, n0.properties)::nodecomposi with s0 as (select e0.id as e0, (n0.id, n0.kind_ids, n0.properties)::nodecomposite as n0, (n1.id, n1.kind_ids, n1.properties)::nodecomposite as n1 from edge e0 join node n0 on n0.id = e0.start_id join node n1 on n1.id = e0.end_id) select ((case when (s0.n0).id is null or s0.e0 is null or (s0.n1).id is null then null else ordered_edges_to_path(s0.n0, (select coalesce(array_agg((_edge.id, _edge.start_id, _edge.end_id, _edge.kind_id, _edge.properties)::edgecomposite order by _path.ordinality), array []::edgecomposite[]) from unnest(array [s0.e0]::int8[]) with ordinality as _path(id, ordinality) join edge _edge on _edge.id = _path.id), array [s0.n0, s0.n1]::nodecomposite[])::pathcomposite end).nodes)::nodecomposite[] from s0; -- case: match p = (:NodeKind1)-[:EdgeKind1|EdgeKind2*1..1]->(:NodeKind2) where any(r in relationships(p) where type(r) STARTS WITH 'EdgeKind') return p -with s0 as (with recursive s1_seed(root_id) as not materialized (select n0.id as root_id from node n0 where n0.kind_ids operator (pg_catalog.@>) array [1]::int2[]), s1(root_id, next_id, depth, satisfied, is_cycle, path) as (select e0.start_id, e0.end_id, 1, n1.kind_ids operator (pg_catalog.@>) array [2]::int2[], e0.start_id = e0.end_id, array [e0.id] from s1_seed join edge e0 on e0.start_id = s1_seed.root_id join node n1 on n1.id = e0.end_id where e0.kind_id = any (array [3, 4]::int2[]) union all select s1.root_id, e0.end_id, s1.depth + 1, n1.kind_ids operator (pg_catalog.@>) array [2]::int2[], false, s1.path || e0.id from s1 join lateral (select e0.id, e0.start_id, e0.end_id, e0.kind_id, e0.properties from edge e0 where e0.start_id = s1.next_id and e0.id != all (s1.path) and e0.kind_id = any (array [3, 4]::int2[]) offset 0) e0 on true join node n1 on n1.id = e0.end_id where s1.depth < 1 and not s1.is_cycle) select s1.path as ep0, (n0.id, n0.kind_ids, n0.properties)::nodecomposite as n0, (n1.id, n1.kind_ids, n1.properties)::nodecomposite as n1 from s1 join lateral (select n0.id, n0.kind_ids, n0.properties from node n0 where n0.id = s1.root_id offset 0) n0 on true join lateral (select n1.id, n1.kind_ids, n1.properties from node n1 where n1.id = s1.next_id offset 0) n1 on true where s1.satisfied) select case when (s0.n0).id is null or s0.ep0 is null or (s0.n1).id is null then null else ordered_edges_to_path(s0.n0, (select coalesce(array_agg((_edge.id, _edge.start_id, _edge.end_id, _edge.kind_id, _edge.properties)::edgecomposite order by _path.ordinality), array []::edgecomposite[]) from unnest(s0.ep0) with ordinality as _path(id, ordinality) join edge _edge on _edge.id = _path.id), array [s0.n0, s0.n1]::nodecomposite[])::pathcomposite end as p from s0 where (((select count(*)::int from unnest(((select coalesce(array_agg((_edge.id, _edge.start_id, _edge.end_id, _edge.kind_id, _edge.properties)::edgecomposite order by _path.ordinality), array []::edgecomposite[]) from unnest(s0.ep0) with ordinality as _path(id, ordinality) join edge _edge on _edge.id = _path.id))::edgecomposite[]) as i0 where (kind_name(i0.kind_id)::text like 'EdgeKind%')) >= 1)::bool); +with s0 as (select e0.id as e0, (n0.id, n0.kind_ids, n0.properties)::nodecomposite as n0, (n1.id, n1.kind_ids, n1.properties)::nodecomposite as n1 from edge e0 join node n0 on n0.kind_ids operator (pg_catalog.@>) array [1]::int2[] and n0.id = e0.start_id join node n1 on n1.kind_ids operator (pg_catalog.@>) array [2]::int2[] and n1.id = e0.end_id where e0.kind_id = any (array [3, 4]::int2[])) select case when (s0.n0).id is null or s0.e0 is null or (s0.n1).id is null then null else ordered_edges_to_path(s0.n0, (select coalesce(array_agg((_edge.id, _edge.start_id, _edge.end_id, _edge.kind_id, _edge.properties)::edgecomposite order by _path.ordinality), array []::edgecomposite[]) from unnest(array [s0.e0]::int8[]) with ordinality as _path(id, ordinality) join edge _edge on _edge.id = _path.id), array [s0.n0, s0.n1]::nodecomposite[])::pathcomposite end as p from s0 where ((exists (select 1 from edge i0 where (kind_name(i0.kind_id)::text like 'EdgeKind%') and i0.id = any (array [s0.e0]::int8[])))::bool); + +-- case: match (a)-[*2..2]->(b)-[]->(c) return a +with s0 as (select e0.id as e0, (n0.id, n0.kind_ids, n0.properties)::nodecomposite as n0, (n1.id, n1.kind_ids, n1.properties)::nodecomposite as n1 from edge e0 join node n0 on n0.id = e0.start_id join node n1 on n1.id = e0.end_id), s1 as (select s0.e0 as e0, e1.id as e1, s0.n0 as n0, s0.n1 as n1, (n2.id, n2.kind_ids, n2.properties)::nodecomposite as n2 from s0 join edge e1 on (s0.n1).id = e1.start_id join node n2 on n2.id = e1.end_id where e1.id != s0.e0), s2 as (select s1.e0 as e0, s1.e1 as e1, s1.n0 as n0, s1.n1 as n1, s1.n2 as n2 from s1 join edge e2 on (s1.n2).id = e2.start_id join node n3 on n3.id = e2.end_id where e2.id != s1.e0 and e2.id != s1.e1) select s2.n0 as a from s2; -- case: match p=(:NodeKind1)-[r]->(:NodeKind1) where r.isacl return p limit 100 with s0 as (select (e0.id, e0.start_id, e0.end_id, e0.kind_id, e0.properties)::edgecomposite as e0, (n0.id, n0.kind_ids, n0.properties)::nodecomposite as n0, (n1.id, n1.kind_ids, n1.properties)::nodecomposite as n1 from edge e0 join node n0 on n0.kind_ids operator (pg_catalog.@>) array [1]::int2[] and n0.id = e0.start_id join node n1 on n1.kind_ids operator (pg_catalog.@>) array [1]::int2[] and n1.id = e0.end_id where (((e0.properties ->> 'isacl'))::bool) limit 100) select case when (s0.n0).id is null or (s0.e0).id is null or (s0.n1).id is null then null else (array [s0.n0, s0.n1]::nodecomposite[], array [s0.e0]::edgecomposite[])::pathcomposite end as p from s0 limit 100; diff --git a/cypher/models/pgsql/test/translation_cases/shortest_paths.sql b/cypher/models/pgsql/test/translation_cases/shortest_paths.sql index 91bbce1d..d2439bdd 100644 --- a/cypher/models/pgsql/test/translation_cases/shortest_paths.sql +++ b/cypher/models/pgsql/test/translation_cases/shortest_paths.sql @@ -88,9 +88,9 @@ with s0 as (with s1 as (with s2(root_id, next_id, depth, satisfied, is_cycle, pa -- case: MATCH (g1:Group) MATCH (g2:Group) WHERE g1.name STARTS WITH 'DOMAIN USERS@' AND g2.name STARTS WITH 'DOMAIN ADMINS@' MATCH p=shortestPath((g1)-[:AddAllowedToAct|AddMember|AdminTo|AllExtendedRights|AllowedToDelegate|CanRDP|Contains|ForceChangePassword|GenericAll|GenericWrite|GetChangesAll|GetChanges|HasSession|MemberOf|Owns|ReadLAPSPassword|SQLAdmin|TrustedBy|WriteAccountRestrictions|WriteOwner*1..]->(g2)) WHERE NONE(r IN relationships(p) WHERE type(r) = 'HasSession' AND startNode(r).name = 'DF-WIN10-DEV01.DUMPSTER.FIRE') RETURN p -- pgsql_params:{"pi0":"insert into next_front (root_id, next_id, depth, satisfied, is_cycle, path) with s3_seed(root_id) as not materialized (select s3_seed_filter.id as root_id from traversal_root_filter s3_seed_filter) select e0.start_id, e0.end_id, 1, exists (select 1 from traversal_pair_filter where traversal_pair_filter.root_id = e0.start_id and traversal_pair_filter.terminal_id = e0.end_id), e0.start_id = e0.end_id, array [e0.id] from s3_seed join edge e0 on e0.start_id = s3_seed.root_id where e0.kind_id = any (array [14, 15, 16, 17, 18, 19, 12, 20, 21, 22, 23, 24, 7, 25, 26, 27, 28, 29, 30, 31]::int2[]) and case when (select count(*)::int8 from traversal_terminal_filter where traversal_terminal_filter.id = e0.start_id) = 0 then true else shortest_path_self_endpoint_error(e0.start_id, e0.start_id) end;","pi1":"insert into next_front (root_id, next_id, depth, satisfied, is_cycle, path) select s3.root_id, e0.end_id, s3.depth + 1, exists (select 1 from traversal_pair_filter where traversal_pair_filter.root_id = s3.root_id and traversal_pair_filter.terminal_id = e0.end_id), false, s3.path || e0.id from forward_front s3 join edge e0 on e0.start_id = s3.next_id where e0.kind_id = any (array [14, 15, 16, 17, 18, 19, 12, 20, 21, 22, 23, 24, 7, 25, 26, 27, 28, 29, 30, 31]::int2[]) and e0.id != all (s3.path) and not exists (select 1 from forward_visited where forward_visited.root_id = s3.root_id and forward_visited.id = e0.end_id);","pi2":"insert into next_front (root_id, next_id, depth, satisfied, is_cycle, path) with s3_seed(root_id) as not materialized (select s3_seed_filter.id as root_id from traversal_terminal_filter s3_seed_filter) select e0.end_id, e0.start_id, 1, exists (select 1 from traversal_pair_filter where traversal_pair_filter.root_id = e0.start_id and traversal_pair_filter.terminal_id = e0.end_id), e0.start_id = e0.end_id, array [e0.id] from s3_seed join edge e0 on e0.end_id = s3_seed.root_id where e0.kind_id = any (array [14, 15, 16, 17, 18, 19, 12, 20, 21, 22, 23, 24, 7, 25, 26, 27, 28, 29, 30, 31]::int2[]);","pi3":"insert into next_front (root_id, next_id, depth, satisfied, is_cycle, path) select s3.root_id, e0.start_id, s3.depth + 1, exists (select 1 from traversal_pair_filter where traversal_pair_filter.root_id = e0.start_id and traversal_pair_filter.terminal_id = s3.root_id), false, e0.id || s3.path from backward_front s3 join edge e0 on e0.end_id = s3.next_id where e0.kind_id = any (array [14, 15, 16, 17, 18, 19, 12, 20, 21, 22, 23, 24, 7, 25, 26, 27, 28, 29, 30, 31]::int2[]) and e0.id != all (s3.path) and not exists (select 1 from backward_visited where backward_visited.root_id = s3.root_id and backward_visited.id = e0.start_id);"} -with s0 as (select (n0.id, n0.kind_ids, n0.properties)::nodecomposite as n0 from node n0 where n0.kind_ids operator (pg_catalog.@>) array [13]::int2[]), s1 as (select s0.n0 as n0, (n1.id, n1.kind_ids, n1.properties)::nodecomposite as n1 from s0, node n1 where ((n1.properties ->> 'name') like 'DOMAIN ADMINS@%' and ((s0.n0).properties ->> 'name') like 'DOMAIN USERS@%') and n1.kind_ids operator (pg_catalog.@>) array [13]::int2[]), s2 as (with s3(root_id, next_id, depth, satisfied, is_cycle, path) as (select * from bidirectional_sp_harness(@pi0::text, @pi1::text, @pi2::text, @pi3::text, 15, ('')::text, ('')::text, ('insert into traversal_pair_filter (root_id, terminal_id) select distinct (s1.n0).id, (s1.n1).id from s1 where (s1.n0).id is not null and (s1.n1).id is not null;')::text)) select s3.path as ep0, (n0.id, n0.kind_ids, n0.properties)::nodecomposite as n0, (n1.id, n1.kind_ids, n1.properties)::nodecomposite as n1 from s1, s3 join node n0 on n0.id = s3.root_id join node n1 on n1.id = s3.next_id where (s1.n0).id = s3.root_id and (s1.n1).id = s3.next_id and case when s3.root_id != s3.next_id then true else shortest_path_self_endpoint_error(s3.root_id, s3.next_id) end) select case when (s2.n0).id is null or s2.ep0 is null or (s2.n1).id is null then null else ordered_edges_to_path(s2.n0, (select coalesce(array_agg((_edge.id, _edge.start_id, _edge.end_id, _edge.kind_id, _edge.properties)::edgecomposite order by _path.ordinality), array []::edgecomposite[]) from unnest(s2.ep0) with ordinality as _path(id, ordinality) join edge _edge on _edge.id = _path.id), array [s2.n0, s2.n1]::nodecomposite[])::pathcomposite end as p from s2 where (((select count(*)::int from unnest(((select coalesce(array_agg((_edge.id, _edge.start_id, _edge.end_id, _edge.kind_id, _edge.properties)::edgecomposite order by _path.ordinality), array []::edgecomposite[]) from unnest(s2.ep0) with ordinality as _path(id, ordinality) join edge _edge on _edge.id = _path.id))::edgecomposite[]) as i0 where ((jsonb_typeof(((start_node(i0)::nodecomposite).properties -> 'name')) = 'string' and ((start_node(i0)::nodecomposite).properties ->> 'name') = 'DF-WIN10-DEV01.DUMPSTER.FIRE') and i0.kind_id = 7)) = 0 and ((select coalesce(array_agg((_edge.id, _edge.start_id, _edge.end_id, _edge.kind_id, _edge.properties)::edgecomposite order by _path.ordinality), array []::edgecomposite[]) from unnest(s2.ep0) with ordinality as _path(id, ordinality) join edge _edge on _edge.id = _path.id))::edgecomposite[] is not null)::bool); +with s0 as (select (n0.id, n0.kind_ids, n0.properties)::nodecomposite as n0 from node n0 where n0.kind_ids operator (pg_catalog.@>) array [13]::int2[]), s1 as (select s0.n0 as n0, (n1.id, n1.kind_ids, n1.properties)::nodecomposite as n1 from s0, node n1 where ((n1.properties ->> 'name') like 'DOMAIN ADMINS@%' and ((s0.n0).properties ->> 'name') like 'DOMAIN USERS@%') and n1.kind_ids operator (pg_catalog.@>) array [13]::int2[]), s2 as (with s3(root_id, next_id, depth, satisfied, is_cycle, path) as (select * from bidirectional_sp_harness(@pi0::text, @pi1::text, @pi2::text, @pi3::text, 15, ('')::text, ('')::text, ('insert into traversal_pair_filter (root_id, terminal_id) select distinct (s1.n0).id, (s1.n1).id from s1 where (s1.n0).id is not null and (s1.n1).id is not null;')::text)) select s3.path as ep0, (n0.id, n0.kind_ids, n0.properties)::nodecomposite as n0, (n1.id, n1.kind_ids, n1.properties)::nodecomposite as n1 from s1, s3 join node n0 on n0.id = s3.root_id join node n1 on n1.id = s3.next_id where (s1.n0).id = s3.root_id and (s1.n1).id = s3.next_id and case when s3.root_id != s3.next_id then true else shortest_path_self_endpoint_error(s3.root_id, s3.next_id) end) select case when (s2.n0).id is null or s2.ep0 is null or (s2.n1).id is null then null else ordered_edges_to_path(s2.n0, (select coalesce(array_agg((_edge.id, _edge.start_id, _edge.end_id, _edge.kind_id, _edge.properties)::edgecomposite order by _path.ordinality), array []::edgecomposite[]) from unnest(s2.ep0) with ordinality as _path(id, ordinality) join edge _edge on _edge.id = _path.id), array [s2.n0, s2.n1]::nodecomposite[])::pathcomposite end as p from s2 where ((not exists (select 1 from edge i0 where ((jsonb_typeof(((start_node((i0.id, i0.start_id, i0.end_id, i0.kind_id, i0.properties)::edgecomposite)::nodecomposite).properties -> 'name')) = 'string' and ((start_node((i0.id, i0.start_id, i0.end_id, i0.kind_id, i0.properties)::edgecomposite)::nodecomposite).properties ->> 'name') = 'DF-WIN10-DEV01.DUMPSTER.FIRE') and i0.kind_id = 7) and i0.id = any (s2.ep0)) and s2.ep0 is not null)::bool); -- case: match p=shortestPath((s:NodeKind1)-[:EdgeKind1|HasSession*1..]->(d:NodeKind1)) where s.name = 'path-filter-src' and d.name = 'path-filter-dst' with p where none(r in relationships(p) where type(r) = 'HasSession' and startNode(r).name = 'blocked-session-host') return p -- pgsql_params:{"pi0":"insert into next_front (root_id, next_id, depth, satisfied, is_cycle, path) with s2_seed(root_id) as not materialized (select n0.id as root_id from node n0 where ((jsonb_typeof((n0.properties -\u003e 'name')) = 'string' and (n0.properties -\u003e\u003e 'name') = 'path-filter-src')) and n0.kind_ids operator (pg_catalog.@\u003e) array [1]::int2[]) select e0.start_id, e0.end_id, 1, exists (select 1 from traversal_pair_filter where traversal_pair_filter.root_id = e0.start_id and traversal_pair_filter.terminal_id = e0.end_id), e0.start_id = e0.end_id, array [e0.id] from s2_seed join edge e0 on e0.start_id = s2_seed.root_id where e0.kind_id = any (array [3, 7]::int2[]) and case when (select count(*)::int8 from traversal_pair_filter where traversal_pair_filter.root_id = e0.start_id and traversal_pair_filter.terminal_id = e0.start_id) = 0 then true else shortest_path_self_endpoint_error(e0.start_id, e0.start_id) end;","pi1":"insert into next_front (root_id, next_id, depth, satisfied, is_cycle, path) select s2.root_id, e0.end_id, s2.depth + 1, exists (select 1 from traversal_pair_filter where traversal_pair_filter.root_id = s2.root_id and traversal_pair_filter.terminal_id = e0.end_id), false, s2.path || e0.id from forward_front s2 join edge e0 on e0.start_id = s2.next_id where e0.kind_id = any (array [3, 7]::int2[]) and e0.id != all (s2.path) and not exists (select 1 from forward_visited where forward_visited.root_id = s2.root_id and forward_visited.id = e0.end_id);","pi2":"insert into next_front (root_id, next_id, depth, satisfied, is_cycle, path) with s2_seed(root_id) as not materialized (select n1.id as root_id from node n1 where ((jsonb_typeof((n1.properties -\u003e 'name')) = 'string' and (n1.properties -\u003e\u003e 'name') = 'path-filter-dst')) and n1.kind_ids operator (pg_catalog.@\u003e) array [1]::int2[]) select e0.end_id, e0.start_id, 1, exists (select 1 from traversal_pair_filter where traversal_pair_filter.root_id = e0.start_id and traversal_pair_filter.terminal_id = e0.end_id), e0.start_id = e0.end_id, array [e0.id] from s2_seed join edge e0 on e0.end_id = s2_seed.root_id where e0.kind_id = any (array [3, 7]::int2[]);","pi3":"insert into next_front (root_id, next_id, depth, satisfied, is_cycle, path) select s2.root_id, e0.start_id, s2.depth + 1, exists (select 1 from traversal_pair_filter where traversal_pair_filter.root_id = e0.start_id and traversal_pair_filter.terminal_id = s2.root_id), false, e0.id || s2.path from backward_front s2 join edge e0 on e0.end_id = s2.next_id where e0.kind_id = any (array [3, 7]::int2[]) and e0.id != all (s2.path) and not exists (select 1 from backward_visited where backward_visited.root_id = s2.root_id and backward_visited.id = e0.start_id);"} -with s0 as (with s1 as (with s2(root_id, next_id, depth, satisfied, is_cycle, path) as (select * from bidirectional_sp_harness(@pi0::text, @pi1::text, @pi2::text, @pi3::text, 15, ('')::text, ('')::text, ('insert into traversal_pair_filter (root_id, terminal_id) select distinct n0.id, n1.id from node n0, node n1 where ((jsonb_typeof((n0.properties -> ''name'')) = ''string'' and (n0.properties ->> ''name'') = ''path-filter-src'')) and n0.kind_ids operator (pg_catalog.@>) array [1]::int2[] and ((jsonb_typeof((n1.properties -> ''name'')) = ''string'' and (n1.properties ->> ''name'') = ''path-filter-dst'')) and n1.kind_ids operator (pg_catalog.@>) array [1]::int2[] and n0.id is not null and n1.id is not null;')::text)) select s2.path as ep0, (n0.id, n0.kind_ids, n0.properties)::nodecomposite as n0, (n1.id, n1.kind_ids, n1.properties)::nodecomposite as n1 from s2 join node n0 on n0.id = s2.root_id join node n1 on n1.id = s2.next_id where case when s2.root_id != s2.next_id then true else shortest_path_self_endpoint_error(s2.root_id, s2.next_id) end) select case when (s1.n0).id is null or s1.ep0 is null or (s1.n1).id is null then null else ordered_edges_to_path(s1.n0, (select coalesce(array_agg((_edge.id, _edge.start_id, _edge.end_id, _edge.kind_id, _edge.properties)::edgecomposite order by _path.ordinality), array []::edgecomposite[]) from unnest(s1.ep0) with ordinality as _path(id, ordinality) join edge _edge on _edge.id = _path.id), array [s1.n0, s1.n1]::nodecomposite[])::pathcomposite end as pc0 from s1) select s0.pc0 as p from s0 where (((select count(*)::int from unnest(((s0.pc0).edges)::edgecomposite[]) as i0 where ((jsonb_typeof(((start_node(i0)::nodecomposite).properties -> 'name')) = 'string' and ((start_node(i0)::nodecomposite).properties ->> 'name') = 'blocked-session-host') and i0.kind_id = 7)) = 0 and ((s0.pc0).edges)::edgecomposite[] is not null)::bool); +with s0 as (with s1 as (with s2(root_id, next_id, depth, satisfied, is_cycle, path) as (select * from bidirectional_sp_harness(@pi0::text, @pi1::text, @pi2::text, @pi3::text, 15, ('')::text, ('')::text, ('insert into traversal_pair_filter (root_id, terminal_id) select distinct n0.id, n1.id from node n0, node n1 where ((jsonb_typeof((n0.properties -> ''name'')) = ''string'' and (n0.properties ->> ''name'') = ''path-filter-src'')) and n0.kind_ids operator (pg_catalog.@>) array [1]::int2[] and ((jsonb_typeof((n1.properties -> ''name'')) = ''string'' and (n1.properties ->> ''name'') = ''path-filter-dst'')) and n1.kind_ids operator (pg_catalog.@>) array [1]::int2[] and n0.id is not null and n1.id is not null;')::text)) select s2.path as ep0, (n0.id, n0.kind_ids, n0.properties)::nodecomposite as n0, (n1.id, n1.kind_ids, n1.properties)::nodecomposite as n1 from s2 join node n0 on n0.id = s2.root_id join node n1 on n1.id = s2.next_id where case when s2.root_id != s2.next_id then true else shortest_path_self_endpoint_error(s2.root_id, s2.next_id) end) select case when (s1.n0).id is null or s1.ep0 is null or (s1.n1).id is null then null else ordered_edges_to_path(s1.n0, (select coalesce(array_agg((_edge.id, _edge.start_id, _edge.end_id, _edge.kind_id, _edge.properties)::edgecomposite order by _path.ordinality), array []::edgecomposite[]) from unnest(s1.ep0) with ordinality as _path(id, ordinality) join edge _edge on _edge.id = _path.id), array [s1.n0, s1.n1]::nodecomposite[])::pathcomposite end as pc0 from s1) select s0.pc0 as p from s0 where (((select count(*)::int from unnest(((s0.pc0).edges)::edgecomposite[]) as i0 where ((jsonb_typeof(((start_node((i0.id, i0.start_id, i0.end_id, i0.kind_id, i0.properties)::edgecomposite)::nodecomposite).properties -> 'name')) = 'string' and ((start_node((i0.id, i0.start_id, i0.end_id, i0.kind_id, i0.properties)::edgecomposite)::nodecomposite).properties ->> 'name') = 'blocked-session-host') and i0.kind_id = 7)) = 0 and ((s0.pc0).edges)::edgecomposite[] is not null)::bool); diff --git a/cypher/models/pgsql/translate/function.go b/cypher/models/pgsql/translate/function.go index 942e6d22..319ac897 100644 --- a/cypher/models/pgsql/translate/function.go +++ b/cypher/models/pgsql/translate/function.go @@ -602,6 +602,41 @@ func translateNodeLabelsExpression(identifier pgsql.Identifier) pgsql.TypeHinted }, pgsql.TextArray) } +func (s *Translator) relationshipEndpointFunctionArgument(argument pgsql.Expression) pgsql.Expression { + identifier, isIdentifier := unwrapParenthetical(argument).(pgsql.Identifier) + if !isIdentifier { + return argument + } + + if binding, bound := s.scope.Lookup(identifier); bound && bindingExpressionType(binding) == pgsql.EdgeComposite { + return edgeCompositeValue(identifier) + } + + if binding, bound := s.scope.AliasedLookup(identifier); bound && bindingExpressionType(binding) == pgsql.EdgeComposite { + return edgeCompositeValue(identifier) + } + + return argument +} + +func (s *Translator) translateRelationshipEndpointFunction(function pgsql.Identifier, functionInvocation *cypher.FunctionInvocation) error { + if functionInvocation.NumArguments() != 1 { + return fmt.Errorf("expected only one argument for cypher function: %s", functionInvocation.Name) + } + + if argument, err := s.treeTranslator.PopOperand(); err != nil { + return err + } else { + s.treeTranslator.PushOperand(pgsql.FunctionCall{ + Function: function, + Parameters: []pgsql.Expression{s.relationshipEndpointFunctionArgument(argument)}, + CastType: pgsql.NodeComposite, + }) + } + + return nil +} + func (s *Translator) translateFunction(typedExpression *cypher.FunctionInvocation) { switch formattedName := strings.ToLower(typedExpression.Name); formattedName { case cypher.DurationFunction: @@ -669,29 +704,13 @@ func (s *Translator) translateFunction(typedExpression *cypher.FunctionInvocatio } case cypher.StartNodeFunction: - if typedExpression.NumArguments() != 1 { - s.SetError(fmt.Errorf("expected only one argument for cypher function: %s", typedExpression.Name)) - } else if argument, err := s.treeTranslator.PopOperand(); err != nil { + if err := s.translateRelationshipEndpointFunction(pgsql.FunctionStartNode, typedExpression); err != nil { s.SetError(err) - } else { - s.treeTranslator.PushOperand(pgsql.FunctionCall{ - Function: pgsql.FunctionStartNode, - Parameters: []pgsql.Expression{argument}, - CastType: pgsql.NodeComposite, - }) } case cypher.EndNodeFunction: - if typedExpression.NumArguments() != 1 { - s.SetError(fmt.Errorf("expected only one argument for cypher function: %s", typedExpression.Name)) - } else if argument, err := s.treeTranslator.PopOperand(); err != nil { + if err := s.translateRelationshipEndpointFunction(pgsql.FunctionEndNode, typedExpression); err != nil { s.SetError(err) - } else { - s.treeTranslator.PushOperand(pgsql.FunctionCall{ - Function: pgsql.FunctionEndNode, - Parameters: []pgsql.Expression{argument}, - CastType: pgsql.NodeComposite, - }) } case cypher.NodeLabelsFunction: diff --git a/cypher/models/pgsql/translate/function_test.go b/cypher/models/pgsql/translate/function_test.go index 42242991..066f0cf0 100644 --- a/cypher/models/pgsql/translate/function_test.go +++ b/cypher/models/pgsql/translate/function_test.go @@ -123,6 +123,49 @@ func TestProjectionStagesRepeatedPathComponents(t *testing.T) { require.Contains(t, formatted, ".edges") } +func TestRelationshipEndpointFunctionsUseEdgeCompositeArguments(t *testing.T) { + t.Parallel() + + kindMapper := pgutil.NewInMemoryKindMapper() + + query, err := frontend.ParseCypher(frontend.NewContext(), `MATCH ()-[r]->() RETURN startNode(r), endNode(r)`) + require.NoError(t, err) + + translation, err := Translate(context.Background(), query, kindMapper, nil, DefaultGraphID) + require.NoError(t, err) + + formatted, err := Translated(translation) + require.NoError(t, err) + normalized := strings.Join(strings.Fields(formatted), " ") + + require.Contains(t, normalized, "start_node(((s0.e0).id, (s0.e0).start_id, (s0.e0).end_id, (s0.e0).kind_id, (s0.e0).properties)::edgecomposite)") + require.Contains(t, normalized, "end_node(((s0.e0).id, (s0.e0).start_id, (s0.e0).end_id, (s0.e0).kind_id, (s0.e0).properties)::edgecomposite)") + require.NotContains(t, normalized, "start_node(s0.e0)") + require.NotContains(t, normalized, "end_node(s0.e0)") +} + +func TestPathRelationshipPredicateEndpointFunctionUsesEdgeCompositeArguments(t *testing.T) { + t.Parallel() + + query, err := frontend.ParseCypher(frontend.NewContext(), ` +MATCH p = shortestPath((s:Group)-[:MemberOf*1..]->(d:Group)) +WHERE NONE(r IN relationships(p) WHERE type(r) = 'MemberOf' AND startNode(r).name = 'blocked-session-host') +RETURN p +`) + require.NoError(t, err) + + translation, err := Translate(context.Background(), query, optimizerSafetyKindMapper(), nil, DefaultGraphID) + require.NoError(t, err) + + formatted, err := Translated(translation) + require.NoError(t, err) + normalized := strings.Join(strings.Fields(formatted), " ") + + require.Contains(t, normalized, "from edge i0") + require.Contains(t, normalized, "start_node((i0.id, i0.start_id, i0.end_id, i0.kind_id, i0.properties)::edgecomposite)") + require.NotContains(t, normalized, "start_node(i0)") +} + func TestPrepareCollectExpressionMissingBindingErrorNamesArgument(t *testing.T) { t.Parallel() diff --git a/cypher/models/pgsql/translate/match.go b/cypher/models/pgsql/translate/match.go index 2201f4cb..55c955e0 100644 --- a/cypher/models/pgsql/translate/match.go +++ b/cypher/models/pgsql/translate/match.go @@ -35,6 +35,9 @@ func (s *Translator) translateMatch(match *cypher.Match) error { if err := s.buildPatternPredicates(); err != nil { return err } + if err := s.buildPathEdgeIDArrayFutures(); err != nil { + return err + } // If there is no valid previous frame, skip translating an `OPTIONAL MATCH`/treat as plain `MATCH` if match.Optional { diff --git a/cypher/models/pgsql/translate/model.go b/cypher/models/pgsql/translate/model.go index a9254060..9226a72c 100644 --- a/cypher/models/pgsql/translate/model.go +++ b/cypher/models/pgsql/translate/model.go @@ -424,6 +424,8 @@ type ProjectionPruningApplication struct { type TraversalStep struct { Frame *Frame + SourceTarget optimize.TraversalStepTarget + HasSourceTarget bool Direction graph.Direction Expansion *Expansion PathReversed bool @@ -503,6 +505,22 @@ type PatternPart struct { TraversalSteps []*TraversalStep NodeSelect NodeSelect Constraints *ConstraintTracker + nextSourceStep int +} + +func (s *PatternPart) nextSourceTarget() (optimize.TraversalStepTarget, bool) { + if s == nil { + return optimize.TraversalStepTarget{}, false + } + + stepIndex := s.nextSourceStep + s.nextSourceStep++ + + if !s.HasTarget { + return optimize.TraversalStepTarget{}, false + } + + return s.Target.TraversalStep(stepIndex), true } func (s *PatternPart) LastStep() *TraversalStep { @@ -571,6 +589,7 @@ type QueryPart struct { // repetition of some of the exported fields above which is intentional and may be a good refactor target // in the future patternPredicates []*pgsql.Future[*Pattern] + pathEdgeIDArrayFutures []*pgsql.Future[*BoundIdentifier] properties TranslatedProperties currentPattern *Pattern stashedPattern *Pattern @@ -581,6 +600,8 @@ type QueryPart struct { referencedIdentifiers *pgsql.IdentifierSet stashedExpressionTreeTranslator *ExpressionTreeTranslator stashedQuantifierArray []pgsql.Expression + stashedQuantifierUseExists bool + quantifierIndex int quantifierIdentifiers *pgsql.IdentifierSet unwindClauses []UnwindClause isCreating bool @@ -664,6 +685,10 @@ func (s *QueryPart) AddPatternPredicateFuture(predicateFuture *pgsql.Future[*Pat s.patternPredicates = append(s.patternPredicates, predicateFuture) } +func (s *QueryPart) AddPathEdgeIDArrayFuture(pathEdgeIDArrayFuture *pgsql.Future[*BoundIdentifier]) { + s.pathEdgeIDArrayFutures = append(s.pathEdgeIDArrayFutures, pathEdgeIDArrayFuture) +} + func (s *QueryPart) ConsumeCurrentPattern() *Pattern { currentPattern := s.currentPattern s.currentPattern = &Pattern{} diff --git a/cypher/models/pgsql/translate/optimizer_safety_test.go b/cypher/models/pgsql/translate/optimizer_safety_test.go index 0369d50c..5e1786a2 100644 --- a/cypher/models/pgsql/translate/optimizer_safety_test.go +++ b/cypher/models/pgsql/translate/optimizer_safety_test.go @@ -599,6 +599,154 @@ RETURN p require.Contains(t, normalizedQuery, "join edge e0 on e0.end_id = s1_seed.root_id") } +func TestOptimizerSafetyExactOneHopRangeUsesFixedTraversal(t *testing.T) { + t.Parallel() + + translation := optimizerSafetyTranslation(t, ` +MATCH p = (a:Group)-[:MemberOf*1..1]->(b:Group) +WHERE a.name = 'src' +RETURN p + `) + formattedQuery, err := Translated(translation) + require.NoError(t, err) + normalizedQuery := strings.Join(strings.Fields(strings.ToLower(formattedQuery)), " ") + + requirePlannedOptimizationLowering(t, translation.Optimization, optimize.LoweringExactRangeExpansion) + requireOptimizationLowering(t, translation.Optimization, optimize.LoweringExactRangeExpansion) + requirePlannedOptimizationLowering(t, translation.Optimization, optimize.LoweringLatePathMaterialization) + requireOptimizationLowering(t, translation.Optimization, optimize.LoweringLatePathMaterialization) + require.NotContains(t, normalizedQuery, "with recursive") + require.NotContains(t, normalizedQuery, "_seed") + require.Contains(t, normalizedQuery, "from edge e0 join node n0") + require.Contains(t, normalizedQuery, "join node n1") +} + +func TestOptimizerSafetyExactTwoHopRangeUsesFixedTraversal(t *testing.T) { + t.Parallel() + + translation := optimizerSafetyTranslation(t, ` +MATCH p = (a:Group)-[:MemberOf*2..2]->(b:Group) +WHERE a.name = 'src' +RETURN p + `) + formattedQuery, err := Translated(translation) + require.NoError(t, err) + normalizedQuery := strings.Join(strings.Fields(strings.ToLower(formattedQuery)), " ") + + requirePlannedOptimizationLowering(t, translation.Optimization, optimize.LoweringExactRangeExpansion) + requireOptimizationLowering(t, translation.Optimization, optimize.LoweringExactRangeExpansion) + require.NotContains(t, normalizedQuery, "with recursive") + require.NotContains(t, normalizedQuery, "_seed") + require.Contains(t, normalizedQuery, "from edge e0") + require.Contains(t, normalizedQuery, "join edge e1") + require.Contains(t, normalizedQuery, "e1.id !=") + require.Contains(t, normalizedQuery, "array [") +} + +func TestOptimizerSafetyExactTwoHopRangePreservesLaterSourceStepTargets(t *testing.T) { + t.Parallel() + + translation := optimizerSafetyTranslation(t, ` +MATCH (a)-[:MemberOf*2..2]->(b)-[:Enroll]->(c) +RETURN a + `) + formattedQuery, err := Translated(translation) + require.NoError(t, err) + normalizedQuery := strings.Join(strings.Fields(formattedQuery), " ") + + requirePlannedOptimizationLowering(t, translation.Optimization, optimize.LoweringExactRangeExpansion) + requireOptimizationLowering(t, translation.Optimization, optimize.LoweringExactRangeExpansion) + require.Contains(t, normalizedQuery, "on (s1.n2).id = e2.start_id") + require.NotContains(t, normalizedQuery, "on n2.id = e2.start_id") +} + +func TestOptimizerSafetyExactTwoHopRangeKeepsSyntheticIntermediateNode(t *testing.T) { + t.Parallel() + + normalizedQuery := strings.ToLower(optimizerSafetySQL(t, ` +MATCH (a)-[:MemberOf*2..2]->(b) +RETURN a + `)) + + require.Contains(t, normalizedQuery, "on (s0.n1).id = e1.start_id") + require.NotContains(t, normalizedQuery, "on n1.id = e1.start_id") +} + +func TestOptimizerSafetyConsecutiveExactRangesUseSourceStepTargets(t *testing.T) { + t.Parallel() + + translation := optimizerSafetyTranslation(t, ` +MATCH p = (a)-[:MemberOf*2..2]->(b)-[:Enroll*1..1]->(c) +RETURN p + `) + formattedQuery, err := Translated(translation) + require.NoError(t, err) + normalizedQuery := strings.Join(strings.Fields(strings.ToLower(formattedQuery)), " ") + + requirePlannedOptimizationLowering(t, translation.Optimization, optimize.LoweringExactRangeExpansion) + requireOptimizationLowering(t, translation.Optimization, optimize.LoweringExactRangeExpansion) + requireNoSkippedOptimizationLowering(t, translation.Optimization, optimize.LoweringExactRangeExpansion) + require.NotContains(t, normalizedQuery, "with recursive") + require.NotContains(t, normalizedQuery, "_seed") + require.Contains(t, normalizedQuery, "join edge e2") +} + +func TestOptimizerSafetyExactRangePrefixPreservesSuffixPushdownTargets(t *testing.T) { + t.Parallel() + + translation := optimizerSafetyTranslation(t, ` +MATCH p = (a)-[:MemberOf*2..2]->(b)-[:AdminTo*1..]->(c)-[:Enroll]->(d) +RETURN p + `) + + requirePlannedOptimizationLowering(t, translation.Optimization, optimize.LoweringExactRangeExpansion) + requireOptimizationLowering(t, translation.Optimization, optimize.LoweringExactRangeExpansion) + requirePlannedOptimizationLowering(t, translation.Optimization, optimize.LoweringExpansionSuffixPushdown) + requireOptimizationLowering(t, translation.Optimization, optimize.LoweringExpansionSuffixPushdown) + requireNoSkippedOptimizationLowering(t, translation.Optimization, optimize.LoweringExpansionSuffixPushdown) +} + +func TestOptimizerSafetyPathRelationshipPredicateUsesPathIDs(t *testing.T) { + t.Parallel() + + translation := optimizerSafetyTranslation(t, ` +MATCH p = (a:Group)-[:MemberOf*1..]->(b:Group) +WHERE any(r in relationships(p) WHERE type(r) STARTS WITH 'Member') +RETURN p + `) + formattedQuery, err := Translated(translation) + require.NoError(t, err) + normalizedQuery := strings.Join(strings.Fields(strings.ToLower(formattedQuery)), " ") + + requirePlannedOptimizationLowering(t, translation.Optimization, optimize.LoweringPathRelationshipPredicate) + requireOptimizationLowering(t, translation.Optimization, optimize.LoweringPathRelationshipPredicate) + require.Contains(t, normalizedQuery, "exists (select 1 from edge i0 where") + require.Contains(t, normalizedQuery, "i0.id = any (s0.ep0)") + require.Contains(t, normalizedQuery, "kind_name(i0.kind_id)::text like 'member%'") + require.NotContains(t, normalizedQuery, "select count(*)::int from unnest") + require.NotContains(t, normalizedQuery, "from unnest(((select coalesce(array_agg") +} + +func TestOptimizerSafetyNonePathRelationshipPredicateUsesNotExists(t *testing.T) { + t.Parallel() + + translation := optimizerSafetyTranslation(t, ` +MATCH p = (a:Group)-[:MemberOf*1..]->(b:Group) +WHERE none(r in relationships(p) WHERE type(r) = 'AdminTo') +RETURN p + `) + formattedQuery, err := Translated(translation) + require.NoError(t, err) + normalizedQuery := strings.Join(strings.Fields(strings.ToLower(formattedQuery)), " ") + + requirePlannedOptimizationLowering(t, translation.Optimization, optimize.LoweringPathRelationshipPredicate) + requireOptimizationLowering(t, translation.Optimization, optimize.LoweringPathRelationshipPredicate) + require.Contains(t, normalizedQuery, "not exists (select 1 from edge i0 where") + require.Contains(t, normalizedQuery, "i0.id = any (s0.ep0)") + require.Contains(t, normalizedQuery, "s0.ep0 is not null") + require.NotContains(t, normalizedQuery, "select count(*)::int from unnest") +} + func TestOptimizerSafetyAggregateTraversalCountUsesIDOnlySourceAnchoredShape(t *testing.T) { t.Parallel() diff --git a/cypher/models/pgsql/translate/path_functions.go b/cypher/models/pgsql/translate/path_functions.go index 516f3022..ad2e77e9 100644 --- a/cypher/models/pgsql/translate/path_functions.go +++ b/cypher/models/pgsql/translate/path_functions.go @@ -40,6 +40,56 @@ func pathCompositeEdgesExpression(scope *Scope, pathBinding *BoundIdentifier) (p return pgsql.ArrayLiteral{CastType: pgsql.EdgeCompositeArray}, nil } +func pathCompositeEdgeIDArrayExpression(scope *Scope, pathBinding *BoundIdentifier) (pgsql.Expression, error) { + var edgeIDArrayReferences []pgsql.Expression + + for _, dependency := range pathBinding.Dependencies { + switch dependency.DataType { + case pgsql.ExpansionPath: + edgeIDArrayReferences = append(edgeIDArrayReferences, pathBindingReference(scope, dependency)) + + case pgsql.EdgeComposite: + edgeIDArrayReferences = append(edgeIDArrayReferences, pgsql.ArrayLiteral{ + Values: []pgsql.Expression{ + pathCompositeColumnReference(scope, dependency, pgsql.ColumnID), + }, + CastType: pgsql.Int8Array, + }) + + case pgsql.PathEdge: + edgeIDArrayReferences = append(edgeIDArrayReferences, pgsql.ArrayLiteral{ + Values: []pgsql.Expression{ + pathEdgeIDReference(scope, dependency), + }, + CastType: pgsql.Int8Array, + }) + + default: + // Path bindings also depend on their node endpoints. Those are not part of relationships(p). + } + } + + if edgeIDArrayExpression := concatenatePathCompositeParts(edgeIDArrayReferences); edgeIDArrayExpression != nil { + return edgeIDArrayExpression, nil + } + + return pgsql.ArrayLiteral{ + CastType: pgsql.Int8Array, + }, nil +} + +func (s *Translator) buildPathEdgeIDArrayFutures() error { + for _, future := range s.query.CurrentPart().pathEdgeIDArrayFutures { + if edgeIDArrayExpression, err := pathCompositeEdgeIDArrayExpression(s.scope, future.Data); err != nil { + return err + } else { + future.SyntaxNode = edgeIDArrayExpression + } + } + + return nil +} + func resolvePathCompositeFieldReference(scope *Scope, reference pgsql.RowColumnReference) (pgsql.Expression, bool, error) { identifier, isIdentifier := unwrapParenthetical(reference.Identifier).(pgsql.Identifier) if !isIdentifier { diff --git a/cypher/models/pgsql/translate/predicate_placement.go b/cypher/models/pgsql/translate/predicate_placement.go index 6ed48f32..e1dbfde1 100644 --- a/cypher/models/pgsql/translate/predicate_placement.go +++ b/cypher/models/pgsql/translate/predicate_placement.go @@ -6,11 +6,16 @@ import ( ) func (s *Translator) recordPredicatePlacementConsumption(part *PatternPart, stepIndex int, traversalStep *TraversalStep, constraints PatternConstraints) { - if part == nil || !part.HasTarget || traversalStep == nil { + if traversalStep == nil { return } - for _, decision := range s.predicatePlacementDecisions[part.Target.TraversalStep(stepIndex)] { + target, hasTarget := sourceTargetForTraversalStep(part, stepIndex) + if !hasTarget { + return + } + + for _, decision := range s.predicatePlacementDecisions[target] { if predicatePlacementDecisionConsumed(decision, traversalStep, constraints) { s.recordLowering(optimize.LoweringPredicatePlacement) return diff --git a/cypher/models/pgsql/translate/projection.go b/cypher/models/pgsql/translate/projection.go index af3eff99..d83f7a4a 100644 --- a/cypher/models/pgsql/translate/projection.go +++ b/cypher/models/pgsql/translate/projection.go @@ -191,6 +191,26 @@ func pathCompositeReference(scope *Scope, binding *BoundIdentifier, columns []pg } } +func edgeCompositeValue(expression pgsql.Expression) pgsql.CompositeValue { + value := pgsql.CompositeValue{ + DataType: pgsql.EdgeComposite, + } + + for _, edgeTableColumn := range pgsql.EdgeTableColumns { + switch typedExpression := expression.(type) { + case pgsql.Identifier: + value.Values = append(value.Values, pgsql.CompoundIdentifier{typedExpression, edgeTableColumn}) + default: + value.Values = append(value.Values, pgsql.RowColumnReference{ + Identifier: expression, + Column: edgeTableColumn, + }) + } + } + + return value +} + func pathCompositeColumnReference(scope *Scope, binding *BoundIdentifier, column pgsql.Identifier) pgsql.Expression { if binding.LastProjection != nil || scope.CurrentFrameBinding() != nil { return pgsql.RowColumnReference{ @@ -486,18 +506,10 @@ func buildProjectionForEdgeComposite(alias pgsql.Identifier, projected *BoundIde }, nil } - value := pgsql.CompositeValue{ - DataType: pgsql.EdgeComposite, - } - - for _, edgeTableColumn := range pgsql.EdgeTableColumns { - value.Values = append(value.Values, pgsql.CompoundIdentifier{projected.Identifier, edgeTableColumn}) - } - // Create a new final projection that's aliased to the visible binding's identifier return []pgsql.SelectItem{ &pgsql.AliasedExpression{ - Expression: value, + Expression: edgeCompositeValue(projected.Identifier), Alias: pgsql.AsOptionalIdentifier(alias), }, }, nil diff --git a/cypher/models/pgsql/translate/quantifiers.go b/cypher/models/pgsql/translate/quantifiers.go index ca4c7178..96b79af3 100644 --- a/cypher/models/pgsql/translate/quantifiers.go +++ b/cypher/models/pgsql/translate/quantifiers.go @@ -6,8 +6,40 @@ import ( "github.com/specterops/dawgs/cypher/models" "github.com/specterops/dawgs/cypher/models/cypher" "github.com/specterops/dawgs/cypher/models/pgsql" + "github.com/specterops/dawgs/cypher/models/pgsql/optimize" ) +func (s *Translator) enterQuantifier() { + if !s.query.HasParts() { + return + } + + currentPart := s.query.CurrentPart() + target := optimize.QuantifierTarget{ + QueryPartIndex: len(s.query.Parts) - 1, + QuantifierIndex: currentPart.quantifierIndex, + } + + currentPart.quantifierIndex++ + s.quantifierTargets = append(s.quantifierTargets, target) +} + +func (s *Translator) exitQuantifier() { + if len(s.quantifierTargets) == 0 { + return + } + + s.quantifierTargets = s.quantifierTargets[:len(s.quantifierTargets)-1] +} + +func (s *Translator) currentQuantifierTarget() (optimize.QuantifierTarget, bool) { + if len(s.quantifierTargets) == 0 { + return optimize.QuantifierTarget{}, false + } + + return s.quantifierTargets[len(s.quantifierTargets)-1], true +} + func (s *Translator) prepareFilterExpression(filterExpression *cypher.FilterExpression) error { quantifierExpressionTree := NewExpressionTreeTranslator(s.kindMapper) @@ -36,9 +68,14 @@ func (s *Translator) prepareFilterExpression(filterExpression *cypher.FilterExpr func (s *Translator) translateFilterExpression(filterExpression *cypher.FilterExpression) error { var ( - iDInColFromClause = s.query.CurrentPart().ConsumeFromClauses()[0] + currentPart = s.query.CurrentPart() + fromClauses = currentPart.ConsumeFromClauses() ) + if len(fromClauses) == 0 { + return fmt.Errorf("filter expression has no collection source") + } + if identifier, hasCypherBinding, err := extractIdentifierFromCypherExpression(filterExpression.Specifier); err != nil { return err } else if !hasCypherBinding { @@ -52,19 +89,36 @@ func (s *Translator) translateFilterExpression(filterExpression *cypher.FilterEx if constraints, err := s.treeTranslator.ConsumeAllConstraints(); err != nil { return err } else { - nestedQuery := &pgsql.Parenthetical{ - Expression: pgsql.Select{ - Projection: []pgsql.SelectItem{ - pgsql.FunctionCall{ - Function: pgsql.FunctionCount, - Parameters: []pgsql.Expression{pgsql.WildcardIdentifier}, - CastType: pgsql.Int, + var nestedQuery pgsql.Expression + + if currentPart.stashedQuantifierUseExists { + nestedQuery = pgsql.ExistsExpression{ + Subquery: pgsql.Subquery{ + Query: pgsql.Query{ + Body: pgsql.Select{ + Projection: []pgsql.SelectItem{pgsql.NewLiteral(1, pgsql.Int)}, + From: fromClauses, + Where: constraints.Expression.AsExpression(), + }, }, }, - From: []pgsql.FromClause{iDInColFromClause}, - Where: constraints.Expression.AsExpression(), - }, + } + } else { + nestedQuery = &pgsql.Parenthetical{ + Expression: pgsql.Select{ + Projection: []pgsql.SelectItem{ + pgsql.FunctionCall{ + Function: pgsql.FunctionCount, + Parameters: []pgsql.Expression{pgsql.WildcardIdentifier}, + CastType: pgsql.Int, + }, + }, + From: fromClauses, + Where: constraints.Expression.AsExpression(), + }, + } } + // push nested query on stashed expression tree translator s.query.CurrentPart().stashedExpressionTreeTranslator.treeBuilder.PushOperand(nestedQuery) s.treeTranslator = s.query.CurrentPart().stashedExpressionTreeTranslator @@ -74,6 +128,76 @@ func (s *Translator) translateFilterExpression(filterExpression *cypher.FilterEx return nil } +func relationshipsPathIdentifier(expression pgsql.Expression) (pgsql.Identifier, bool) { + switch typedExpression := unwrapParenthetical(expression).(type) { + case pgsql.TypeCast: + return relationshipsPathIdentifier(typedExpression.Expression) + + case pgsql.RowColumnReference: + if typedExpression.Column != pgsql.ColumnEdges { + return "", false + } + + identifier, isIdentifier := unwrapParenthetical(typedExpression.Identifier).(pgsql.Identifier) + return identifier, isIdentifier + + default: + return "", false + } +} + +func (s *Translator) translatePathRelationshipPredicateSource(array *BoundIdentifier, quantifierArrayExpression pgsql.Expression) (bool, error) { + if s.query.CurrentPart().currentPattern == nil || s.query.CurrentPart().currentPattern.Parts == nil { + return false, nil + } + + target, hasTarget := s.currentQuantifierTarget() + if !hasTarget { + return false, nil + } + + if _, hasDecision := s.pathRelationshipPredicateDecisions[target]; !hasDecision { + return false, nil + } + + pathIdentifier, isRelationshipsPath := relationshipsPathIdentifier(quantifierArrayExpression) + if !isRelationshipsPath { + return false, nil + } + + pathBinding, bound := pathCompositeBinding(s.scope, pathIdentifier) + if !bound { + return false, nil + } + + edgeIDs := pgsql.NewFuture(pathBinding, pgsql.Int8Array) + + array.DataType = pgsql.EdgeComposite + s.query.CurrentPart().stashedQuantifierArray = []pgsql.Expression{edgeIDs} + s.query.CurrentPart().stashedQuantifierUseExists = true + s.query.CurrentPart().AddPathEdgeIDArrayFuture(edgeIDs) + s.query.CurrentPart().AddFromClause(pgsql.FromClause{ + Source: pgsql.TableReference{ + Name: pgsql.TableEdge.AsCompoundIdentifier(), + Binding: pgsql.AsOptionalIdentifier(array.Identifier), + }, + }) + + if err := s.treeTranslator.AddTranslationConstraint( + pgsql.AsIdentifierSet(array.Identifier), + pgsql.NewBinaryExpression( + pgsql.CompoundIdentifier{array.Identifier, pgsql.ColumnID}, + pgsql.OperatorEquals, + pgsql.NewAnyExpression(edgeIDs, pgsql.Int8Array), + ), + ); err != nil { + return true, err + } + + s.recordLowering(optimize.LoweringPathRelationshipPredicate) + return true, nil +} + func (s *Translator) translateQuantifierArrayExpression(quantifierArrayExpression pgsql.Expression) ([]pgsql.Expression, pgsql.DataType, error) { switch typedQuantifierArrayExpression := quantifierArrayExpression.(type) { case *pgsql.BinaryExpression: @@ -134,6 +258,12 @@ func (s *Translator) translateIDInCollection(idInCol *cypher.IDInCollection) err ) quantifierArrayExpression := quantifierArray.AsExpression() + if lowered, err := s.translatePathRelationshipPredicateSource(array, quantifierArrayExpression); err != nil { + return err + } else if lowered { + return nil + } + if translatedParameters, inferredCollectionType, err := s.translateQuantifierArrayExpression(quantifierArrayExpression); err != nil { return err } else { @@ -169,6 +299,39 @@ func (s *Translator) buildQuantifier(cypherQuantifierExpression *cypher.Quantifi if filterExpression, err := s.treeTranslator.PopOperand(); err != nil { s.SetError(err) } else { + if s.query.CurrentPart().stashedQuantifierUseExists { + s.query.CurrentPart().stashedQuantifierUseExists = false + + switch cypherQuantifierExpression.Type { + case cypher.QuantifierTypeAny: + s.treeTranslator.PushOperand(pgsql.NewTypeCast(filterExpression, pgsql.Boolean)) + return nil + + case cypher.QuantifierTypeNone: + negatedExpression := filterExpression + if existsExpression, isExistsExpression := filterExpression.(pgsql.ExistsExpression); isExistsExpression { + existsExpression.Negated = true + negatedExpression = existsExpression + } else { + negatedExpression = pgsql.NewUnaryExpression(pgsql.OperatorNot, filterExpression) + } + + s.treeTranslator.PushOperand(pgsql.NewTypeCast(pgsql.NewBinaryExpression( + negatedExpression, + pgsql.OperatorAnd, + pgsql.NewBinaryExpression( + s.query.CurrentPart().stashedQuantifierArray[0], + pgsql.OperatorIsNot, + pgsql.NullLiteral(), + ), + ), pgsql.Boolean)) + return nil + + default: + return fmt.Errorf("path relationship predicate lowering does not support quantifier type: %v", cypherQuantifierExpression.Type) + } + } + switch cypherQuantifierExpression.Type { case cypher.QuantifierTypeAny: quantifierExpression = pgsql.Literal{Value: 1, CastType: pgsql.Int} diff --git a/cypher/models/pgsql/translate/relationship.go b/cypher/models/pgsql/translate/relationship.go index 872583d8..51ff6381 100644 --- a/cypher/models/pgsql/translate/relationship.go +++ b/cypher/models/pgsql/translate/relationship.go @@ -5,6 +5,7 @@ import ( "github.com/specterops/dawgs/cypher/models/cypher" "github.com/specterops/dawgs/cypher/models/pgsql" + "github.com/specterops/dawgs/cypher/models/pgsql/optimize" ) func (s *Translator) translateRelationshipPattern(relationshipPattern *cypher.RelationshipPattern) error { @@ -18,15 +19,19 @@ func (s *Translator) translateRelationshipPattern(relationshipPattern *cypher.Re } else if currentQueryPart.isCreating { return s.collectCreateEdgePattern(relationshipPattern, patternPart, bindingResult) } else { - if err := s.translateRelationshipPatternToStep(bindingResult, patternPart, relationshipPattern); err != nil { + edgeBindings, err := s.translateRelationshipPatternToStep(bindingResult, patternPart, relationshipPattern) + if err != nil { return err } if currentQueryPart.HasProperties() { - if propertyConstraints, err := s.buildPatternPropertyConstraints(bindingResult.Binding, currentQueryPart.ConsumeProperties()); err != nil { - return err - } else if err := s.treeTranslator.AddTranslationConstraint(pgsql.NewIdentifierSet().Add(bindingResult.Binding.Identifier), propertyConstraints); err != nil { - return err + properties := currentQueryPart.ConsumeProperties() + for _, edgeBinding := range edgeBindings { + if propertyConstraints, err := s.buildPatternPropertyConstraints(edgeBinding, properties); err != nil { + return err + } else if err := s.treeTranslator.AddTranslationConstraint(pgsql.NewIdentifierSet().Add(edgeBinding.Identifier), propertyConstraints); err != nil { + return err + } } } @@ -34,12 +39,16 @@ func (s *Translator) translateRelationshipPattern(relationshipPattern *cypher.Re if len(relationshipPattern.Kinds) > 0 { if kindIDs, err := s.kindMapper.MapKinds(relationshipPattern.Kinds); err != nil { return fmt.Errorf("failed to translate kinds: %w", err) - } else if err := s.treeTranslator.AddTranslationConstraint(pgsql.NewIdentifierSet().Add(bindingResult.Binding.Identifier), pgsql.NewBinaryExpression( - pgsql.CompoundIdentifier{bindingResult.Binding.Identifier, pgsql.ColumnKindID}, - pgsql.OperatorEquals, - pgsql.NewAnyExpressionHinted(pgsql.NewLiteral(kindIDs, pgsql.Int2Array)), - )); err != nil { - return err + } else { + for _, edgeBinding := range edgeBindings { + if err := s.treeTranslator.AddTranslationConstraint(pgsql.NewIdentifierSet().Add(edgeBinding.Identifier), pgsql.NewBinaryExpression( + pgsql.CompoundIdentifier{edgeBinding.Identifier, pgsql.ColumnKindID}, + pgsql.OperatorEquals, + pgsql.NewAnyExpressionHinted(pgsql.NewLiteral(kindIDs, pgsql.Int2Array)), + )); err != nil { + return err + } + } } } } @@ -90,12 +99,105 @@ func (s *Translator) collectCreateEdgePattern(relationshipPattern *cypher.Relati return nil } -func (s *Translator) translateRelationshipPatternToStep(bindingResult BindingResult, part *PatternPart, relationshipPattern *cypher.RelationshipPattern) error { +func (s *Translator) exactRangeExpansionDecision(sourceTarget optimize.TraversalStepTarget, hasSourceTarget bool, relationshipPattern *cypher.RelationshipPattern) (optimize.ExactRangeExpansionDecision, bool) { + if !hasSourceTarget || relationshipPattern == nil { + return optimize.ExactRangeExpansionDecision{}, false + } + + decision, hasDecision := s.exactRangeExpansionDecisions[sourceTarget] + if !hasDecision || decision.Depth != optimize.ExactPatternRangeDepth(relationshipPattern.Range) { + return optimize.ExactRangeExpansionDecision{}, false + } + + return decision, true +} + +func (s *Translator) translateExactRangeRelationshipPatternToSteps( + firstEdge *BoundIdentifier, + part *PatternPart, + relationshipPattern *cypher.RelationshipPattern, + isContinuation bool, + currentStep *TraversalStep, + depth int64, + sourceTarget optimize.TraversalStepTarget, + hasSourceTarget bool, +) ([]*BoundIdentifier, error) { + if depth < 1 { + return nil, fmt.Errorf("expected exact range depth >= 1 but found %d", depth) + } + + edgeBindings := []*BoundIdentifier{firstEdge} + var firstStep *TraversalStep + + if isContinuation { + firstStep = &TraversalStep{ + Edge: firstEdge, + Direction: relationshipPattern.Direction, + LeftNode: currentStep.RightNode, + LeftNodeBound: true, + SourceTarget: sourceTarget, + HasSourceTarget: hasSourceTarget, + } + part.TraversalSteps = append(part.TraversalSteps, firstStep) + } else { + firstStep = currentStep + firstStep.Edge = firstEdge + firstStep.Direction = relationshipPattern.Direction + firstStep.SourceTarget = sourceTarget + firstStep.HasSourceTarget = hasSourceTarget + } + + if part.PatternBinding != nil { + part.PatternBinding.DependOn(firstEdge) + } + + previousStep := firstStep + for hop := int64(1); hop < depth; hop++ { + intermediateNode, err := s.scope.DefineNew(pgsql.NodeComposite) + if err != nil { + return nil, err + } + + previousStep.RightNode = intermediateNode + previousStep.RightNodeBound = false + + if part.PatternBinding != nil { + part.PatternBinding.DependOn(intermediateNode) + } + + nextEdge, err := s.scope.DefineNew(pgsql.EdgeComposite) + if err != nil { + return nil, err + } + + edgeBindings = append(edgeBindings, nextEdge) + if part.PatternBinding != nil { + part.PatternBinding.DependOn(nextEdge) + } + + nextStep := &TraversalStep{ + Edge: nextEdge, + Direction: relationshipPattern.Direction, + LeftNode: intermediateNode, + LeftNodeBound: true, + SourceTarget: sourceTarget, + HasSourceTarget: hasSourceTarget, + } + part.TraversalSteps = append(part.TraversalSteps, nextStep) + previousStep = nextStep + } + + s.recordLowering(optimize.LoweringExactRangeExpansion) + return edgeBindings, nil +} + +func (s *Translator) translateRelationshipPatternToStep(bindingResult BindingResult, part *PatternPart, relationshipPattern *cypher.RelationshipPattern) ([]*BoundIdentifier, error) { var ( - expansion *Expansion - numSteps = len(part.TraversalSteps) - currentStep = part.TraversalSteps[numSteps-1] - isContinuation = currentStep.Edge != nil + expansion *Expansion + numSteps = len(part.TraversalSteps) + currentStep = part.TraversalSteps[numSteps-1] + isContinuation = currentStep.Edge != nil + sourceTarget, hasSourceTarget = part.nextSourceTarget() ) if bindingResult.AlreadyBound { @@ -103,8 +205,10 @@ func (s *Translator) translateRelationshipPatternToStep(bindingResult BindingRes // This is a traversal continuation so copy the right node identifier of the preceding step and then // add the new step nextStep := &TraversalStep{ - Edge: bindingResult.Binding, - Direction: relationshipPattern.Direction, + Edge: bindingResult.Binding, + Direction: relationshipPattern.Direction, + SourceTarget: sourceTarget, + HasSourceTarget: hasSourceTarget, } // Mark the left node as already bound as it's part of the previous step's continuation @@ -116,13 +220,28 @@ func (s *Translator) translateRelationshipPatternToStep(bindingResult BindingRes // Carry over the left node identifier if the edge identifier for the preceding step isn't set currentStep.Edge = bindingResult.Binding currentStep.Direction = relationshipPattern.Direction + currentStep.SourceTarget = sourceTarget + currentStep.HasSourceTarget = hasSourceTarget } - return nil + return []*BoundIdentifier{bindingResult.Binding}, nil } // Look for any relationship pattern ranges. These indicate some kind of variable expansion of the path pattern. if relationshipPattern.Range != nil { + if decision, shouldLower := s.exactRangeExpansionDecision(sourceTarget, hasSourceTarget, relationshipPattern); shouldLower { + return s.translateExactRangeRelationshipPatternToSteps( + bindingResult.Binding, + part, + relationshipPattern, + isContinuation, + currentStep, + decision.Depth, + sourceTarget, + hasSourceTarget, + ) + } + // Set the edge type to an expansion of edges bindingResult.Binding.DataType = pgsql.ExpansionEdge @@ -133,7 +252,7 @@ func (s *Translator) translateRelationshipPatternToStep(bindingResult BindingRes } if expansionPathBinding, err := s.scope.DefineNew(pgsql.ExpansionPath); err != nil { - return err + return nil, err } else { // Set up the new expansion model here expansion = NewExpansionModel(part, relationshipPattern) @@ -157,18 +276,22 @@ func (s *Translator) translateRelationshipPatternToStep(bindingResult BindingRes // This is a traversal continuation so copy the right node identifier of the preceding step and then // add the new step part.TraversalSteps = append(part.TraversalSteps, &TraversalStep{ - Edge: bindingResult.Binding, - Direction: relationshipPattern.Direction, - LeftNode: currentStep.RightNode, - LeftNodeBound: true, - Expansion: expansion, + Edge: bindingResult.Binding, + Direction: relationshipPattern.Direction, + LeftNode: currentStep.RightNode, + LeftNodeBound: true, + Expansion: expansion, + SourceTarget: sourceTarget, + HasSourceTarget: hasSourceTarget, }) } else { // Carry over the left node identifier if the edge identifier for the preceding step isn't set currentStep.Edge = bindingResult.Binding currentStep.Direction = relationshipPattern.Direction currentStep.Expansion = expansion + currentStep.SourceTarget = sourceTarget + currentStep.HasSourceTarget = hasSourceTarget } - return nil + return []*BoundIdentifier{bindingResult.Binding}, nil } diff --git a/cypher/models/pgsql/translate/renamer.go b/cypher/models/pgsql/translate/renamer.go index 0b80a499..f516cbbc 100644 --- a/cypher/models/pgsql/translate/renamer.go +++ b/cypher/models/pgsql/translate/renamer.go @@ -92,6 +92,20 @@ func (s *FrameBindingRewriter) rewriteArraySlice(slice *pgsql.ArraySlice) error return nil } +func (s *FrameBindingRewriter) rewriteArrayLiteral(literal *pgsql.ArrayLiteral) error { + if literal == nil { + return nil + } + + for idx := range literal.Values { + if err := s.rewriteExpression(&literal.Values[idx]); err != nil { + return err + } + } + + return nil +} + func (s *FrameBindingRewriter) rewriteExpression(expression *pgsql.Expression) error { if expression == nil || *expression == nil { return nil @@ -120,6 +134,15 @@ func (s *FrameBindingRewriter) rewriteExpression(expression *pgsql.Expression) e case *pgsql.ArraySlice: return s.rewriteArraySlice(typedExpression) + + case pgsql.ArrayLiteral: + if err := s.rewriteArrayLiteral(&typedExpression); err != nil { + return err + } + *expression = typedExpression + + case *pgsql.ArrayLiteral: + return s.rewriteArrayLiteral(typedExpression) } return nil @@ -214,6 +237,17 @@ func (s *FrameBindingRewriter) enter(node pgsql.SyntaxNode) error { if err := s.rewriteArraySlice(typedValue); err != nil { return err } + + case pgsql.ArrayLiteral: + if err := s.rewriteArrayLiteral(&typedValue); err != nil { + return err + } + typedExpression.Values[idx] = typedValue + + case *pgsql.ArrayLiteral: + if err := s.rewriteArrayLiteral(typedValue); err != nil { + return err + } } } @@ -265,6 +299,17 @@ func (s *FrameBindingRewriter) enter(node pgsql.SyntaxNode) error { if err := s.rewriteArraySlice(typedParameter); err != nil { return err } + + case pgsql.ArrayLiteral: + if err := s.rewriteArrayLiteral(&typedParameter); err != nil { + return err + } + typedExpression.Parameters[idx] = typedParameter + + case *pgsql.ArrayLiteral: + if err := s.rewriteArrayLiteral(typedParameter); err != nil { + return err + } } } @@ -359,6 +404,12 @@ func (s *FrameBindingRewriter) enter(node pgsql.SyntaxNode) error { case *pgsql.ArraySlice: return s.rewriteArraySlice(typedExpression) + case pgsql.ArrayLiteral: + return s.rewriteArrayLiteral(&typedExpression) + + case *pgsql.ArrayLiteral: + return s.rewriteArrayLiteral(typedExpression) + case *pgsql.Parenthetical: switch typedInnerExpression := typedExpression.Expression.(type) { case pgsql.Identifier: @@ -385,6 +436,17 @@ func (s *FrameBindingRewriter) enter(node pgsql.SyntaxNode) error { if err := s.rewriteArraySlice(typedInnerExpression); err != nil { return err } + + case pgsql.ArrayLiteral: + if err := s.rewriteArrayLiteral(&typedInnerExpression); err != nil { + return err + } + typedExpression.Expression = typedInnerExpression + + case *pgsql.ArrayLiteral: + if err := s.rewriteArrayLiteral(typedInnerExpression); err != nil { + return err + } } case *pgsql.EdgeArrayFromPathIDs: @@ -416,6 +478,17 @@ func (s *FrameBindingRewriter) enter(node pgsql.SyntaxNode) error { if err := s.rewriteArraySlice(typedInnerExpression); err != nil { return err } + + case pgsql.ArrayLiteral: + if err := s.rewriteArrayLiteral(&typedInnerExpression); err != nil { + return err + } + typedExpression.Expression = typedInnerExpression + + case *pgsql.ArrayLiteral: + if err := s.rewriteArrayLiteral(typedInnerExpression); err != nil { + return err + } } case *pgsql.AnyExpression: @@ -444,6 +517,17 @@ func (s *FrameBindingRewriter) enter(node pgsql.SyntaxNode) error { if err := s.rewriteArraySlice(typedInnerExpression); err != nil { return err } + + case pgsql.ArrayLiteral: + if err := s.rewriteArrayLiteral(&typedInnerExpression); err != nil { + return err + } + typedExpression.Expression = typedInnerExpression + + case *pgsql.ArrayLiteral: + if err := s.rewriteArrayLiteral(typedInnerExpression); err != nil { + return err + } } case *pgsql.UnaryExpression: @@ -472,6 +556,17 @@ func (s *FrameBindingRewriter) enter(node pgsql.SyntaxNode) error { if err := s.rewriteArraySlice(typedOperand); err != nil { return err } + + case pgsql.ArrayLiteral: + if err := s.rewriteArrayLiteral(&typedOperand); err != nil { + return err + } + typedExpression.Operand = typedOperand + + case *pgsql.ArrayLiteral: + if err := s.rewriteArrayLiteral(typedOperand); err != nil { + return err + } } case *pgsql.BinaryExpression: @@ -500,6 +595,17 @@ func (s *FrameBindingRewriter) enter(node pgsql.SyntaxNode) error { if err := s.rewriteArraySlice(typedLOperand); err != nil { return err } + + case pgsql.ArrayLiteral: + if err := s.rewriteArrayLiteral(&typedLOperand); err != nil { + return err + } + typedExpression.LOperand = typedLOperand + + case *pgsql.ArrayLiteral: + if err := s.rewriteArrayLiteral(typedLOperand); err != nil { + return err + } } switch typedROperand := typedExpression.ROperand.(type) { @@ -527,6 +633,17 @@ func (s *FrameBindingRewriter) enter(node pgsql.SyntaxNode) error { if err := s.rewriteArraySlice(typedROperand); err != nil { return err } + + case pgsql.ArrayLiteral: + if err := s.rewriteArrayLiteral(&typedROperand); err != nil { + return err + } + typedExpression.ROperand = typedROperand + + case *pgsql.ArrayLiteral: + if err := s.rewriteArrayLiteral(typedROperand); err != nil { + return err + } } } diff --git a/cypher/models/pgsql/translate/renamer_test.go b/cypher/models/pgsql/translate/renamer_test.go index 50b349e0..ad653412 100644 --- a/cypher/models/pgsql/translate/renamer_test.go +++ b/cypher/models/pgsql/translate/renamer_test.go @@ -70,6 +70,23 @@ func TestRewriteFrameBindings(t *testing.T) { Expected: &pgsql.EdgeArrayFromPathIDs{ PathIDs: rewrittenA, }, + }, { + Case: pgsql.NewBinaryExpression( + a.Identifier, + pgsql.OperatorEquals, + pgsql.NewAnyExpression(pgsql.ArrayLiteral{ + Values: []pgsql.Expression{a.Identifier}, + CastType: pgsql.IntArray, + }, pgsql.IntArray), + ), + Expected: pgsql.NewBinaryExpression( + rewrittenA, + pgsql.OperatorEquals, + pgsql.NewAnyExpression(pgsql.ArrayLiteral{ + Values: []pgsql.Expression{rewrittenA}, + CastType: pgsql.IntArray, + }, pgsql.IntArray), + ), }, { Case: pgsql.NewBinaryExpression( pgsql.ArraySlice{ diff --git a/cypher/models/pgsql/translate/translator.go b/cypher/models/pgsql/translate/translator.go index b300fcdf..8a53e5c1 100644 --- a/cypher/models/pgsql/translate/translator.go +++ b/cypher/models/pgsql/translate/translator.go @@ -33,19 +33,22 @@ type Translator struct { collectIDMembershipAliases map[pgsql.Identifier]struct{} collectIDProjectionDepth int - appliedLoweringCounts map[string]int - patternTargets map[*cypher.PatternPart]optimize.PatternTarget - patternPredicateTargets map[*cypher.PatternPredicate]optimize.PatternTarget - projectionPruningDecisions map[optimize.TraversalStepTarget]optimize.ProjectionPruningDecision - latePathDecisions map[optimize.TraversalStepTarget][]optimize.LatePathMaterializationDecision - suffixPushdownDecisions map[optimize.TraversalStepTarget][]optimize.ExpansionSuffixPushdownDecision - predicatePlacementDecisions map[optimize.TraversalStepTarget][]optimize.PredicatePlacementDecision - expandIntoDecisions map[optimize.TraversalStepTarget]optimize.ExpandIntoDecision - traversalDirectionDecisions map[optimize.TraversalStepTarget]optimize.TraversalDirectionDecision - shortestPathStrategyDecisions map[optimize.TraversalStepTarget]optimize.ShortestPathStrategyDecision - shortestPathFilterDecisions map[optimize.TraversalStepTarget][]optimize.ShortestPathFilterDecision - limitPushdownDecisions map[optimize.TraversalStepTarget][]optimize.LimitPushdownDecision - patternPredicateDecisions map[optimize.TraversalStepTarget]optimize.PatternPredicatePlacementDecision + appliedLoweringCounts map[string]int + patternTargets map[*cypher.PatternPart]optimize.PatternTarget + patternPredicateTargets map[*cypher.PatternPredicate]optimize.PatternTarget + projectionPruningDecisions map[optimize.TraversalStepTarget]optimize.ProjectionPruningDecision + latePathDecisions map[optimize.TraversalStepTarget][]optimize.LatePathMaterializationDecision + suffixPushdownDecisions map[optimize.TraversalStepTarget][]optimize.ExpansionSuffixPushdownDecision + predicatePlacementDecisions map[optimize.TraversalStepTarget][]optimize.PredicatePlacementDecision + expandIntoDecisions map[optimize.TraversalStepTarget]optimize.ExpandIntoDecision + traversalDirectionDecisions map[optimize.TraversalStepTarget]optimize.TraversalDirectionDecision + shortestPathStrategyDecisions map[optimize.TraversalStepTarget]optimize.ShortestPathStrategyDecision + shortestPathFilterDecisions map[optimize.TraversalStepTarget][]optimize.ShortestPathFilterDecision + limitPushdownDecisions map[optimize.TraversalStepTarget][]optimize.LimitPushdownDecision + patternPredicateDecisions map[optimize.TraversalStepTarget]optimize.PatternPredicatePlacementDecision + exactRangeExpansionDecisions map[optimize.TraversalStepTarget]optimize.ExactRangeExpansionDecision + pathRelationshipPredicateDecisions map[optimize.QuantifierTarget]optimize.PathRelationshipPredicateDecision + quantifierTargets []optimize.QuantifierTarget } func NewTranslator(ctx context.Context, kindMapper pgsql.KindMapper, parameters map[string]any, graphID int32) *Translator { @@ -92,6 +95,8 @@ func (s *Translator) SetOptimizationPlan(plan optimize.Plan) { s.shortestPathFilterDecisions = map[optimize.TraversalStepTarget][]optimize.ShortestPathFilterDecision{} s.limitPushdownDecisions = map[optimize.TraversalStepTarget][]optimize.LimitPushdownDecision{} s.patternPredicateDecisions = map[optimize.TraversalStepTarget]optimize.PatternPredicatePlacementDecision{} + s.exactRangeExpansionDecisions = map[optimize.TraversalStepTarget]optimize.ExactRangeExpansionDecision{} + s.pathRelationshipPredicateDecisions = map[optimize.QuantifierTarget]optimize.PathRelationshipPredicateDecision{} for _, decision := range plan.LoweringPlan.ProjectionPruning { s.projectionPruningDecisions[decision.Target] = decision @@ -132,6 +137,14 @@ func (s *Translator) SetOptimizationPlan(plan optimize.Plan) { for _, decision := range plan.LoweringPlan.PatternPredicate { s.patternPredicateDecisions[decision.Target] = decision } + + for _, decision := range plan.LoweringPlan.ExactRangeExpansion { + s.exactRangeExpansionDecisions[decision.Target] = decision + } + + for _, decision := range plan.LoweringPlan.PathRelationshipPredicate { + s.pathRelationshipPredicateDecisions[decision.Target] = decision + } } func (s *Translator) Enter(expression cypher.SyntaxNode) { @@ -144,7 +157,10 @@ func (s *Translator) Enter(expression cypher.SyntaxNode) { *cypher.FunctionInvocation, *cypher.Order, *cypher.RemoveItem, *cypher.SetItem, *cypher.MapItem, *cypher.UpdatingClause, *cypher.Delete, *cypher.With, *cypher.Return, *cypher.MultiPartQuery, *cypher.Properties, *cypher.KindMatcher, - *cypher.Quantifier, *cypher.IDInCollection: + *cypher.IDInCollection: + + case *cypher.Quantifier: + s.enterQuantifier() case *cypher.RangeQuantifier: if typedExpression.Value != string(pgsql.WildcardIdentifier) { @@ -388,6 +404,7 @@ func (s *Translator) Exit(expression cypher.SyntaxNode) { if err := s.buildQuantifier(typedExpression); err != nil { s.SetError(err) } + s.exitQuantifier() case *cypher.NodePattern: if err := s.translateNodePattern(typedExpression); err != nil { @@ -699,17 +716,58 @@ func (s *Translator) recordSkippedLowerings() { func plannedLoweringCounts(plan optimize.LoweringPlan) []SkippedLowering { return []SkippedLowering{ - {Name: optimize.LoweringProjectionPruning, Count: len(plan.ProjectionPruning)}, - {Name: optimize.LoweringLatePathMaterialization, Count: len(plan.LatePathMaterialization)}, - {Name: optimize.LoweringExpandIntoDetection, Count: len(plan.ExpandInto)}, - {Name: optimize.LoweringTraversalDirection, Count: len(plan.TraversalDirection)}, - {Name: optimize.LoweringShortestPathStrategy, Count: len(plan.ShortestPathStrategy)}, - {Name: optimize.LoweringShortestPathFilter, Count: len(plan.ShortestPathFilter)}, - {Name: optimize.LoweringLimitPushdown, Count: len(plan.LimitPushdown)}, - {Name: optimize.LoweringExpansionSuffixPushdown, Count: len(plan.ExpansionSuffixPushdown)}, - {Name: optimize.LoweringPredicatePlacement, Count: len(plan.PredicatePlacement) + len(plan.PatternPredicate)}, - {Name: optimize.LoweringCountStoreFastPath, Count: len(plan.CountStoreFastPath)}, - {Name: optimize.LoweringAggregateTraversalCount, Count: len(plan.AggregateTraversalCount)}, + { + Name: optimize.LoweringProjectionPruning, + Count: len(plan.ProjectionPruning), + }, + { + Name: optimize.LoweringLatePathMaterialization, + Count: len(plan.LatePathMaterialization), + }, + { + Name: optimize.LoweringExpandIntoDetection, + Count: len(plan.ExpandInto), + }, + { + Name: optimize.LoweringTraversalDirection, + Count: len(plan.TraversalDirection), + }, + { + Name: optimize.LoweringShortestPathStrategy, + Count: len(plan.ShortestPathStrategy), + }, + { + Name: optimize.LoweringShortestPathFilter, + Count: len(plan.ShortestPathFilter), + }, + { + Name: optimize.LoweringLimitPushdown, + Count: len(plan.LimitPushdown), + }, + { + Name: optimize.LoweringExpansionSuffixPushdown, + Count: len(plan.ExpansionSuffixPushdown), + }, + { + Name: optimize.LoweringPredicatePlacement, + Count: len(plan.PredicatePlacement) + len(plan.PatternPredicate), + }, + { + Name: optimize.LoweringCountStoreFastPath, + Count: len(plan.CountStoreFastPath), + }, + { + Name: optimize.LoweringExactRangeExpansion, + Count: len(plan.ExactRangeExpansion), + }, + { + Name: optimize.LoweringPathRelationshipPredicate, + Count: len(plan.PathRelationshipPredicate), + }, + { + Name: optimize.LoweringAggregateTraversalCount, + Count: len(plan.AggregateTraversalCount), + }, } } diff --git a/cypher/models/pgsql/translate/traversal.go b/cypher/models/pgsql/translate/traversal.go index 07495d53..71701f59 100644 --- a/cypher/models/pgsql/translate/traversal.go +++ b/cypher/models/pgsql/translate/traversal.go @@ -27,13 +27,49 @@ func boundEndpointInequality(frame *Frame, traversalStep *TraversalStep) pgsql.E ) } +func sourceTargetForTraversalStep(part *PatternPart, stepIndex int) (optimize.TraversalStepTarget, bool) { + if part == nil || stepIndex < 0 || stepIndex >= len(part.TraversalSteps) { + return optimize.TraversalStepTarget{}, false + } + + if traversalStep := part.TraversalSteps[stepIndex]; traversalStep != nil && traversalStep.HasSourceTarget { + return traversalStep.SourceTarget, true + } + + if !part.HasTarget { + return optimize.TraversalStepTarget{}, false + } + + return part.Target.TraversalStep(stepIndex), true +} + +func traversalStepIsFirstForSourceTarget(part *PatternPart, stepIndex int) bool { + target, hasTarget := sourceTargetForTraversalStep(part, stepIndex) + if !hasTarget || stepIndex == 0 { + return true + } + + previousTarget, previousHasTarget := sourceTargetForTraversalStep(part, stepIndex-1) + return !previousHasTarget || previousTarget != target +} + +func traversalStepIsLastForSourceTarget(part *PatternPart, stepIndex int) bool { + target, hasTarget := sourceTargetForTraversalStep(part, stepIndex) + if !hasTarget || stepIndex+1 >= len(part.TraversalSteps) { + return true + } + + nextTarget, nextHasTarget := sourceTargetForTraversalStep(part, stepIndex+1) + return !nextHasTarget || nextTarget != target +} + func (s *Translator) shouldUseExpandInto(part *PatternPart, stepIndex int, traversalStep *TraversalStep) bool { if traversalStep == nil || traversalStep.Expansion != nil || !traversalStep.LeftNodeBound || !traversalStep.RightNodeBound { return false } - if part != nil && part.HasTarget { - if _, hasDecision := s.expandIntoDecisions[part.Target.TraversalStep(stepIndex)]; hasDecision { + if target, hasTarget := sourceTargetForTraversalStep(part, stepIndex); hasTarget { + if _, hasDecision := s.expandIntoDecisions[target]; hasDecision { return true } @@ -44,11 +80,12 @@ func (s *Translator) shouldUseExpandInto(part *PatternPart, stepIndex int, trave } func (s *Translator) traversalDirectionDecision(part *PatternPart, stepIndex int) (optimize.TraversalDirectionDecision, bool) { - if part == nil || !part.HasTarget { + target, hasTarget := sourceTargetForTraversalStep(part, stepIndex) + if !hasTarget { return optimize.TraversalDirectionDecision{}, false } - decision, hasDecision := s.traversalDirectionDecisions[part.Target.TraversalStep(stepIndex)] + decision, hasDecision := s.traversalDirectionDecisions[target] return decision, hasDecision } @@ -81,11 +118,12 @@ func (s *Translator) applyPatternConstraintBalance(part *PatternPart, stepIndex } func (s *Translator) shortestPathStrategyDecision(part *PatternPart, stepIndex int) (optimize.ShortestPathStrategyDecision, bool) { - if part == nil || !part.HasTarget { + target, hasTarget := sourceTargetForTraversalStep(part, stepIndex) + if !hasTarget { return optimize.ShortestPathStrategyDecision{}, false } - decision, hasDecision := s.shortestPathStrategyDecisions[part.Target.TraversalStep(stepIndex)] + decision, hasDecision := s.shortestPathStrategyDecisions[target] return decision, hasDecision } @@ -116,11 +154,12 @@ func (s *Translator) useBidirectionalShortestPathStrategy(part *PatternPart, ste } func (s *Translator) shortestPathFilterDecisionsForStep(part *PatternPart, stepIndex int) []optimize.ShortestPathFilterDecision { - if part == nil || !part.HasTarget { + target, hasTarget := sourceTargetForTraversalStep(part, stepIndex) + if !hasTarget { return nil } - return s.shortestPathFilterDecisions[part.Target.TraversalStep(stepIndex)] + return s.shortestPathFilterDecisions[target] } func (s *Translator) applyShortestPathFilterMaterialization(part *PatternPart, stepIndex int, traversalStep *TraversalStep, expansionModel *Expansion) { @@ -142,11 +181,12 @@ func (s *Translator) applyShortestPathFilterMaterialization(part *PatternPart, s } func (s *Translator) hasLimitPushdownDecision(part *PatternPart, stepIndex int, mode optimize.LimitPushdownMode) bool { - if part == nil || !part.HasTarget { + target, hasTarget := sourceTargetForTraversalStep(part, stepIndex) + if !hasTarget { return true } - for _, decision := range s.limitPushdownDecisions[part.Target.TraversalStep(stepIndex)] { + for _, decision := range s.limitPushdownDecisions[target] { if decision.Mode == mode { return true } @@ -638,10 +678,12 @@ func (s *Translator) applyExpansionSuffixPushdown(part *PatternPart) (int, error var applied int for stepIndex := range part.TraversalSteps { - var ( - target = part.Target.TraversalStep(stepIndex) - decisions = s.suffixPushdownDecisions[target] - ) + target, hasTarget := sourceTargetForTraversalStep(part, stepIndex) + if !hasTarget { + continue + } + + decisions := s.suffixPushdownDecisions[target] if len(decisions) == 0 { continue @@ -649,16 +691,40 @@ func (s *Translator) applyExpansionSuffixPushdown(part *PatternPart) (int, error for _, decision := range decisions { if decision.SuffixLength <= 0 || - decision.SuffixStartStep <= stepIndex || + decision.SuffixStartStep <= target.StepIndex || decision.SuffixEndStep < decision.SuffixStartStep || - decision.SuffixEndStep >= len(part.TraversalSteps) || decision.SuffixEndStep-decision.SuffixStartStep+1 != decision.SuffixLength { continue } + var ( + suffixStartTarget = target + suffixEndTarget = target + ) + suffixStartTarget.StepIndex = decision.SuffixStartStep + suffixEndTarget.StepIndex = decision.SuffixEndStep + + suffixStartIndex, suffixEndIndex := -1, -1 + for candidateIndex := range part.TraversalSteps { + candidateTarget, candidateHasTarget := sourceTargetForTraversalStep(part, candidateIndex) + if !candidateHasTarget { + continue + } + + if candidateTarget == suffixStartTarget && suffixStartIndex < 0 { + suffixStartIndex = candidateIndex + } + if candidateTarget == suffixEndTarget { + suffixEndIndex = candidateIndex + } + } + if suffixStartIndex < 0 || suffixEndIndex < suffixStartIndex { + continue + } + var ( currentStep = part.TraversalSteps[stepIndex] - suffixSteps = part.TraversalSteps[decision.SuffixStartStep : decision.SuffixEndStep+1] + suffixSteps = part.TraversalSteps[suffixStartIndex : suffixEndIndex+1] ) if candidateApplied, err := applyExpansionSuffixPushdownCandidate(currentStep, suffixSteps); err != nil { @@ -736,11 +802,12 @@ func previousRelationshipUniquenessConstraint(scope *Scope, part *PatternPart, s } func (s *Translator) projectionPruningDecision(part *PatternPart, stepIndex int) (optimize.ProjectionPruningDecision, bool) { - if part == nil || !part.HasTarget { + target, hasTarget := sourceTargetForTraversalStep(part, stepIndex) + if !hasTarget { return optimize.ProjectionPruningDecision{}, false } - decision, hasDecision := s.projectionPruningDecisions[part.Target.TraversalStep(stepIndex)] + decision, hasDecision := s.projectionPruningDecisions[target] return decision, hasDecision } @@ -750,7 +817,7 @@ func (s *Translator) prepareProjectionPruning(part *PatternPart, stepIndex int, return } - if decision.OmitLeftNode { + if decision.OmitLeftNode && traversalStepIsFirstForSourceTarget(part, stepIndex) { traversalStep.ProjectionPruning.LeftNode = traversalStep.LeftNode } @@ -758,7 +825,7 @@ func (s *Translator) prepareProjectionPruning(part *PatternPart, stepIndex int, traversalStep.ProjectionPruning.Relationship = traversalStep.Edge } - if decision.OmitRightNode { + if decision.OmitRightNode && traversalStepIsLastForSourceTarget(part, stepIndex) { traversalStep.ProjectionPruning.RightNode = traversalStep.RightNode } @@ -768,11 +835,12 @@ func (s *Translator) prepareProjectionPruning(part *PatternPart, stepIndex int, } func (s *Translator) latePathMaterializationDecision(part *PatternPart, stepIndex int, mode optimize.LatePathMaterializationMode) (optimize.LatePathMaterializationDecision, bool) { - if part == nil || !part.HasTarget { + target, hasTarget := sourceTargetForTraversalStep(part, stepIndex) + if !hasTarget { return optimize.LatePathMaterializationDecision{}, false } - for _, decision := range s.latePathDecisions[part.Target.TraversalStep(stepIndex)] { + for _, decision := range s.latePathDecisions[target] { if decision.Mode == mode { return decision, true } diff --git a/docs/development.md b/docs/development.md new file mode 100644 index 00000000..b39bcfe7 --- /dev/null +++ b/docs/development.md @@ -0,0 +1,132 @@ +# Development Workflow + +This document covers repository workflows for contributors. The top-level [README](../README.md) keeps the main +quick-start commands and links to detailed documentation. + +## Build And Test + +The [Makefile](../Makefile) drives build, test, formatting, and reporting commands. + +```bash +make build +make test +``` + +`make test` runs unit tests with race detection and writes coverage artifacts under `.coverage/`: + +- `.coverage/unit.out` +- `.coverage/coverage.txt` + +Run the integration suite when a backend is available: + +```bash +export CONNECTION_STRING="postgresql://dawgs:weneedbetterpasswords@localhost:65432/dawgs" +make test_integration +``` + +`CONNECTION_STRING` selects the active backend by URL scheme. Neo4j connection strings may use `neo4j://`, +`neo4j+s://`, or `neo4j+ssc://`; a single path segment selects the Neo4j database name. + +Benign local examples: + +```bash +export CONNECTION_STRING="postgresql://dawgs:weneedbetterpasswords@localhost:65432/dawgs" +export CONNECTION_STRING="neo4j://neo4j:weneedbetterpasswords@localhost:7687" +``` + +Use backend-specific targets when needed: + +```bash +make test_pg +make test_neo4j +``` + +`make test_all` runs unit tests and integration tests: + +```bash +make test_all +``` + +The shared integration cases under `integration/testdata/cases` and `integration/testdata/templates` must stay +semantically equivalent across supported backends. Avoid driver-specific skips or expected results in those files; add +driver-scoped integration coverage when a backend-only capability needs coverage. + +## Formatting + +Run: + +```bash +make format +``` + +The target uses `goimports`; install it locally if it is missing from your environment. + +## Quality And Metrics + +Cyclomatic complexity, CRAP, and quality signal reports are available through dedicated metric targets: + +```bash +make complexity +make crap +make quality +make metrics +``` + +Outputs are written under `.coverage/`: + +- `.coverage/cyclomatic.txt` +- `.coverage/crap.txt` +- `.coverage/crap.json` +- `.coverage/quality.txt` +- `.coverage/quality.json` +- `.coverage/metrics.html` + +Generated parser files, tests, vendor code, and testdata are excluded from these reports. The HTML report embeds its CSS +and JavaScript directly in the document, so it can be opened without network access. + +Optional quality inputs can be supplied through Make variables: + +```bash +make quality BACKEND_RESULT_ARGS="-backend-result pg=.coverage/integration-pg.json -backend-result neo4j=.coverage/integration-neo4j.json" +make quality BENCHMARK_REPORT=.coverage/benchmark.json BENCHMARK_BASELINE=.coverage/benchmark-baseline.json +make quality FUZZ_REPORT=.coverage/fuzz.json MUTATION_REPORT=.coverage/mutation.json +``` + +`make quality_backend` captures PostgreSQL and Neo4j integration results for backend equivalence comparison. It +requires `PG_CONNECTION_STRING` and `NEO4J_CONNECTION_STRING`. `make quality_bench` writes benchmark markdown and JSON +captures for later baseline comparison. + +Thresholds are report-only by default. To enforce configured thresholds, run: + +```bash +make metrics_check +``` + +The defaults can be adjusted with `CYCLO_TOP`, `CYCLO_OVER`, `CRAP_TOP`, `CRAP_OVER`, and `BENCHMARK_REGRESSION`. + +## Plan Corpus + +`make plan_corpus` captures plan diagnostics for the shared Cypher integration corpus. It accepts either +`CONNECTION_STRING` for one backend or `PG_CONNECTION_STRING` and `NEO4J_CONNECTION_STRING` for both backends, then +writes JSONL captures and markdown/JSON summaries under `.coverage/`. + +Run it when changing PostgreSQL Cypher planning, lowering, or SQL emission. The summaries rank expensive PostgreSQL +plans and report recursive CTEs, `SubPlan`, `Function Scan on unnest`, planned/applied optimizer lowerings, and +skipped-lowering reasons. + +See [Plan Corpus Capture](../cmd/plancorpus/README.md) for flags and review guidance. + +## Graph Benchmarks + +`go run ./cmd/graphbench` captures runtime diagnostics for the scale corpus under `benchmark/testdata/scale`. + +Current modes are: + +- `postgres_sql` +- `local_traversal` +- `neo4j` + +AGE is reference-design input only and is not a direct comparison mode. The command can emit JSONL records plus +Markdown and JSON summaries, and can compare current timings against a previous JSONL baseline. + +See [Graph Benchmark Capture](../cmd/graphbench/README.md) for command examples. diff --git a/docs/postgresql_translation.md b/docs/postgresql_translation.md new file mode 100644 index 00000000..c05d2c94 --- /dev/null +++ b/docs/postgresql_translation.md @@ -0,0 +1,87 @@ +# PostgreSQL Translation + +DAWGS translates supported Cypher queries to vanilla PostgreSQL 16 SQL. The implementation lives under +[`cypher/models/pgsql`](../cypher/models/pgsql). + +## Package Layout + +- `format`: PostgreSQL SQL rendering. +- `translate`: openCypher-to-PostgreSQL translation. +- `optimize`: Cypher query-shape analysis and translator lowering decisions. +- `visualization`: PUML graph formatting for PostgreSQL SQL model trees. +- `test`: translation test cases. + +## Optimizer Coverage + +The `optimize` package analyzes Cypher query shape before PostgreSQL SQL emission. Rule outputs are exposed as planned +and applied lowerings in translation diagnostics so plan-corpus captures can catch planning/emission gaps. + +Current PostgreSQL optimization coverage includes: + +- Reproducible plan-corpus capture for PostgreSQL translated SQL, PostgreSQL `EXPLAIN`, Neo4j logical plan operator + trees, planned/applied lowerings, skipped lowerings, and skipped-lowering reasons. +- Count-store fast paths for simple node and directed-edge count queries, including typed variants where kind filters + map cleanly. +- Predicate placement accounting for binding-scope predicates pushed into fixed traversal steps, expansion seeds, + expansion edges, expansion terminal checks, and eligible pattern predicates. +- Shared and late path materialization for path functions such as `nodes(p)`, `relationships(p)`, + `size(relationships(p))`, `startNode`, `endNode`, and `type`. +- Recursive traversal optimizations for endpoint kind/property predicates, relationship type predicates, bound-node + filters, traversal direction selection, and limit pushdown where ordering and distinct semantics permit it. +- Expansion suffix pushdown and `ExpandInto` detection for fixed suffixes and shared-endpoint fanout patterns. +- Strict string property equality lowering through `jsonb_typeof(properties -> key) = 'string'` plus + `properties ->> key = value`, preserving JSON scalar semantics while allowing existing text expression indexes on + selective fields such as `objectid` and `name`. +- Typed relationship count plans that can use the `kind_id`-first covering edge index. +- Correlated relationship `EXISTS` lowering for typed pattern predicates when relationship types and endpoint + correlations are sufficient. +- Membership-only `collect(entity)` ID-array lowering with `id = any(...)` membership predicates. +- Shortest-path strategy and terminal-filter planning for selective endpoint predicates and kind-only terminal filters. +- Exact anonymous directed fixed-range expansion lowering for non-shortest-path `*1..1` and `*2..2` patterns. These + shapes use fixed traversal steps instead of recursive CTEs, preserve path projection semantics, and enforce + relationship uniqueness across emitted fixed steps. The explicit SQL-size cap is depth 2; broader exact ranges + continue through the recursive expansion path. Undirected exact ranges are not eligible for this lowering. +- Predicate-only `ANY`/`NONE` over current path `relationships(p)` bindings lowered to `EXISTS` or `NOT EXISTS` over + path edge IDs, avoiding full `edgecomposite[]` materialization when the final projection does not require it. +- Dependency-safe clause reordering inside non-optional read regions, using existing selectivity heuristics while + preserving stable tie order and pinning clauses with unresolved external dependencies. + +## Indexing Notes + +Exact string property equality is emitted with a JSON string type guard and `properties ->>` extraction. This allows +indexes created on expressions such as `properties ->> 'objectid'` and `properties ->> 'name'` to accelerate selective +anchors without matching JSON booleans or numbers. + +Simple relationship count fast paths depend on the schema's `kind_id`-first edge index for efficient typed counts. + +Substring and suffix predicates are not promoted to blanket schema indexes. PostgreSQL deployments can request explicit +`TextSearchIndex`/trigram property indexes for fields that need `CONTAINS`, `STARTS WITH`, or `ENDS WITH`. Dynamic +parameter/property forms that lower to helper functions remain outside the hard index-match contract until their +lowering changes. + +## Validation Workflow + +Optimizer changes should include focused optimizer/lowering tests, SQL-shape translation tests, and backend-equivalent +integration coverage when behavior affects query semantics. `make test_all` is the default full validation target when +`CONNECTION_STRING` is available. + +Run plan-corpus capture for planner, lowering, or SQL-emission changes: + +```bash +make plan_corpus +``` + +The corpus summary should be checked for PostgreSQL cost, `Recursive Union`, `SubPlan`, `Function Scan on unnest`, and +skipped-lowering deltas. + +PostgreSQL property index regression coverage is hard-failing under the `manual_integration` tag. The synthetic plan +test translates Cypher to PostgreSQL, disables sequential scans for the `EXPLAIN`, and requires explicit node property +indexes to appear in the JSON plan: + +```bash +CONNECTION_STRING="postgresql://dawgs:weneedbetterpasswords@localhost:65432/dawgs" \ + go test -tags manual_integration ./integration -run TestPostgreSQLPropertyIndexPlans +``` + +PostgreSQL-only plan-corpus validation should confirm that `ExactRangeExpansion` and `PathRelationshipPredicate` are +planned and applied for their supported cases without skipped entries for either lowering.