Add Variable expressions

[mhk197]

Summary

This PR adds variables as a first-class Expression and BoundExpression node, and introduces machinery to resolve them against a lexical scope during binding. It is the variable-only foundation for lambdas and higher-order functions.

Variables

A Variable consists only of a name. It participates explicitly in expression traversal, optimization, validity, and analysis as a leaf, similarly to root.

Scope and name resolution

Scope previously represented only the dtype of root(). It now consists of:

• the root dtype
• an ordered stack of lexical Frames.

A frame contains an ordered set of (Variable, DType) bindings introduced by one binder. Duplicate names in the same frame are rejected, while an inner frame may shadow a name from an outer frame.

Scope::push_frame returns an extended scope. Resolution searches from the innermost frame outward and returns both the declared dtype and a stable VariableRef { frame, slot }. Frame indices are counted from the outermost frame and slots follow declaration order, so pushing a new inner frame does not invalidate references to captured variables in outer frames.

For example, a future higher-order function could bind a lambda parameter by extending its surrounding scope:

let lambda_scope = outer_scope.with_bindings([
    (Variable::new("element"), element_dtype),
])?;
let body = var("element").bind_scope(&lambda_scope)?;

Binding

Expression::bind_scope now handles all three expression node kinds:

• Root becomes a bound root carrying scope.root().
• Variable is resolved through the scope and becomes a BoundVariable carrying its source name, declared dtype, and stable VariableRef.
• Scalar recursively binds its children and derives its result dtype from their bound dtypes.

Binding fails if a variable has no enclosing binder. Expression::bind(&root_dtype) remains useful for expressions that only depend on the root, but a variable-containing expression must use bind_scope with the appropriate frames.

Every node in the resulting BoundExpression has a known dtype. In particular, BoundExpression::Variable::dtype() comes from its resolved scope binding rather than from the expression node itself.

Intentional restrictions

• Expression::return_dtype(root_dtype) returns an error for variables. A root dtype alone cannot determine the dtype of a lexical variable; callers must bind against a full Scope.
• ArrayRef::apply rejects unbound variables.
• ArrayRef::apply_bound also rejects bound variables because the current evaluator only receives the root array; it has no runtime environment containing values for VariableRefs yet.

[codspeed-hq]

Merging this PR will regress 2 benchmarks

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 13 improved benchmarks
❌ 2 regressed benchmarks
✅ 2075 untouched benchmarks
⏩ 206 skipped benchmarks¹
🗄️ 4 archived benchmarks run²

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	WallTime	arrow_checked_add_u32_neon[16384]	13.4 µs	20.3 µs	-34.26%
❌	WallTime	mul_u32_nonnull_avx512	5.6 µs	6.3 µs	-10.98%
⚡	WallTime	subtract_shapes_neon[(128, PerRowPerRow)]	3.3 µs	1.9 µs	+73.42%
⚡	WallTime	add_shapes_neon[(128, PerRowPerRow)]	3.2 µs	1.9 µs	+70.86%
⚡	WallTime	multiply_shapes_neon[(128, PerRowPerRow)]	3.2 µs	2 µs	+64.94%
⚡	WallTime	arrow_checked_add_u32_avx512[16384]	21.4 µs	17.6 µs	+21.48%
⚡	WallTime	add_u32_nonnull_neon	7.7 µs	6.5 µs	+18.3%
⚡	WallTime	add_shapes_neon[(16384, PerRowPerRow)]	11.1 µs	9.7 µs	+14.82%
⚡	WallTime	subtract_shapes_neon[(16384, PerRowPerRow)]	11 µs	9.7 µs	+13.8%
⚡	WallTime	words_gather_scalar_avx2[65536]	9.4 µs	8.3 µs	+13.3%
⚡	WallTime	mul_i8_nonnull_neon	12.3 µs	11 µs	+12.46%
⚡	WallTime	add_i64_nullable_neon	12.6 µs	11.3 µs	+11.07%
⚡	WallTime	add_i32_nonnull_neon	8.7 µs	7.9 µs	+11.04%
⚡	WallTime	mul_i16_nonnull_neon	11.4 µs	10.3 µs	+10.93%
⚡	WallTime	add_constant_shapes_neon[(16384, ConstantPerRow)]	10.3 µs	9.3 µs	+10.79%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing mk/variable-expressions (91f7aa1) with develop (47fd3e8)}

Footnotes

1. 206 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

2. 4 benchmarks were run, but are now archived. If they were deleted in another branch, consider rebasing to remove them from the report. Instead if they were added back, click here to restore them. ↩