fix: request a null-aware mark join wherever a NULL mark is observable

[adriangb]

Which issue does this PR close?

• Completes Wrong results: correlated NOT IN with a non-equality correlation returns no rows (null-aware LeftAnti join ignores the residual filter) #25336 for the mark-join shapes.

Important

Stacked on #25558 and #25559. GitHub cannot base a PR on another fork's branch, so the diff here shows all three. Review only the last commit (fix: request a null-aware mark join wherever a NULL mark is observable). Draft until the other two land; I will rebase then.

Rationale for this change

Decorrelation asked for null-aware semantics only for some NOT IN mark joins. A non-equality correlation leaves a residual join filter rather than an equi-join key, and that shape was planned without it, so a NULL on the subquery side read as FALSE instead of UNKNOWN:

CREATE TABLE oc(id INT, g INT, h INT) AS VALUES (2,2,0),(1,2,0),(9,2,5),(3,1,0);
CREATE TABLE ic(id INT, k INT) AS VALUES (1,1),(NULL,1),(2,2);
SELECT id FROM oc WHERE oc.h > 1 OR oc.id NOT IN (SELECT i.id FROM ic i WHERE i.k < oc.g) ORDER BY id;

returns 2, 3, 9; DuckDB returns 3, 9. Row id = 2 is UNKNOWN, not TRUE.

#25559 gives the executor the machinery. This asks for it in the remaining place.

What changes are included in this PR?

A mark join needs null-aware semantics only where a NULL mark can behave differently from a FALSE mark. AND/OR give TRUE only from TRUE, and a Filter keeps a row only when the predicate is TRUE, so a non-negated IN reached through nothing but AND/OR is identical with a FALSE mark and must stay on the plain join. Requesting it there is expensive: 1–2 ms → 7.4 s at 100k × 100k.

That test is made per subquery occurrence, in a recursion that only knows AND/OR. The permissive outcome lives in one arm whose pattern is its own proof — a non-negated IN whose value holds no subquery, reached through nothing but AND/OR frames — and everything else, including future Expr variants, takes the null-aware branch.

Two further changes in build_join:

• a constant IN value is projected as an outer column so the equality becomes on[0], the key position the executor reads as the NOT IN value key. Otherwise a correlation takes that slot and the value-key NULL rules are applied to the wrong key;
• null_aware is computed once instead of being re-derived for the constant projection, the mark branch and the anti branch.

Net effect on the optimizer source is −14 lines.

What is the testing strategy for this PR?

The coverage landed in #25558; this PR flips the remaining expectations, including the Q08 canary, so the diff shows the behaviour change. The plan pins added in #25558 guard the negative direction — that a positive IN stays on the plain join — which no result assertion can catch, since a needless null-aware join is correct, only slower. Mutation testing over the decision confirms it: mutants that widen null-awareness are killed only by those pins.

Are there any user-facing changes?

Correlated NOT IN inside a larger predicate returns correct results.

🤖 Generated with Claude Code

[codecov-commenter]

Codecov Report

❌ Patch coverage is 92.01102% with 29 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.42%. Comparing base (1e09a2a) to head (1c70e49).

Files with missing lines	Patch %	Lines
...fusion/physical-plan/src/joins/hash_join/stream.rs	93.40%	6 Missing and 7 partials ⚠️
...on/optimizer/src/decorrelate_predicate_subquery.rs	86.07%	3 Missing and 8 partials ⚠️
...tafusion/physical-plan/src/joins/hash_join/exec.rs	94.18%	1 Missing and 4 partials ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##             main   #25560    +/-   ##
========================================
  Coverage   82.42%   82.42%            
========================================
  Files        1138     1138            
  Lines      435429   435588   +159     
  Branches   435429   435588   +159     
========================================
+ Hits       358889   359051   +162     
- Misses      54839    54845     +6     
+ Partials    21701    21692     -9     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.