Skip to content

Reject lossy and ambiguous numeric inputs - #27

Merged
tvanreenen merged 3 commits into
mainfrom
input-integrity-0.7.0
Aug 23, 2026
Merged

Reject lossy and ambiguous numeric inputs#27
tvanreenen merged 3 commits into
mainfrom
input-integrity-0.7.0

Conversation

@tvanreenen

Copy link
Copy Markdown
Owner

What this changes

This PR closes two input-conversion gaps before NumPy Vector Store's API becomes stable:

  • A Python within_rows sequence now rejects booleans even when they are mixed with integers. For example, [0, True] raises TypeError instead of becoming row indexes [0, 1] through NumPy dtype coercion.
  • Vector batches and search queries now reject complex-valued inputs with TypeError. Their imaginary components are no longer silently discarded while converting to the store's real float32 representation.

The README documents both rules. The public roadmap now describes 0.7.0 as the final contract-closure release before 1.0, including the later archive-binding, compatibility-policy, and release-hardening work planned for this cycle.

Why this is needed

Both old behaviors accepted data the API never intended to support, and both could change the meaning of a call without telling the caller.

Python and NumPy deliberately treat booleans as integer-like in some conversion contexts. That meant a selector such as [0, True] could bypass the documented rule that booleans are not row indexes. The result depended on whether the boolean appeared alone or beside an integer.

NumPy also permits a complex array to be converted to a real dtype while discarding its imaginary part. A vector such as [1 + 2j, 0] could therefore be stored and searched as [1, 0], producing valid-looking results for different data.

Rejecting these cases at the public boundary is safer than trying to infer caller intent. Because they tighten accidentally accepted inputs, they belong in the pre-1.0 0.7 minor release rather than a patch release.

Implementation details

Python row-selector sequences are inspected before np.asarray() chooses a common dtype. Boolean detection shares the existing single pass used to identify nested shapes, so valid Python selectors are not traversed an extra time. Native NumPy integer arrays keep the current vectorized validation and zero-copy np.intp path.

Vector and query conversion now inspects the input array before requesting float32. The same private conversion boundary serves add() and all three search methods, so complex inputs fail consistently for cosine, dot-product, and Euclidean search. Query validation still runs before empty-result shortcuts, which makes the failure independent of whether the store contains rows.

No public method, return type, archive field, dependency, or valid documented input changes.

User impact

Most users do not need to change anything. Existing real-valued vectors, real-valued queries, and integer row selectors behave as before.

Code that passes Python selectors containing booleans must replace them with the intended integer row indexes or fix the upstream filtering logic. Code that passes complex vectors or queries must choose an explicit real-valued representation before calling the store rather than relying on an implicit lossy cast.

Archive format version 1 remains unchanged, and existing archives require no migration.

Documentation and roadmap

  • Clarifies mixed-boolean selector behavior in the scalar input contract.
  • States that vectors and queries must be real-valued and explains why complex inputs are rejected.
  • Adds the focused 0.7.0 roadmap: lossless input handling, stable archive bindings, an explicit 1.x public contract, and release hardening.
  • Records the features deliberately left outside this stabilization cycle, including update/delete APIs, context-manager persistence, metadata queries, backends, and an archive format change.

Verification

  • uv lock --check
  • ruff check .
  • ruff format --check .
  • mypy src/ benchmarks/
  • 324 tests passed with warnings treated as errors on the locked Python 3.13 environment, with 98% source coverage.
  • 324 tests passed with warnings treated as errors on Python 3.11 and the minimum supported NumPy 1.23.2.
  • Boundary coverage includes Python and NumPy booleans mixed with integers, complex Python sequences, complex NumPy arrays, all three search methods, and both empty and populated stores.
  • An informal one-million-row local check kept native np.intp selector validation near 43 ms; a Python integer list validated near 145 ms on the Apple M4 development machine.

Release note

This is the first PR planned for 0.7.0. It intentionally does not change the package version or changelog yet; those will be prepared after the full 0.7 milestone is complete.

@tvanreenen tvanreenen self-assigned this Aug 22, 2026
@tvanreenen
tvanreenen marked this pull request as ready for review August 23, 2026 01:49
@tvanreenen
tvanreenen merged commit 7caecf1 into main Aug 23, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant