feat: add support for PolarDB for PostgreSQL HNSW - #860
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: cxyybless The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
/assign @XuanYang-cn |
| def __init__(self, *args, drop_old: bool = False, **kwargs): | ||
| super().__init__(*args, drop_old=drop_old, **kwargs) | ||
| self.name = "PolarDBPG" | ||
|
|
There was a problem hiding this comment.
vectordb_bench/backend/clients/polardb_pg/polardb_pg.py line:69
Low ---- self.name is assigned after super().init() returns, so every log line emitted during PgVector.init (config values, create/drop table, create/drop index) and the "config must create an index" RuntimeError message still print "PgVector" for PolarDBPG runs. Consider setting self.name before delegating to the base constructor (or overriding it as a class attribute) so operators can filter PolarDBPG logs consistently.
There was a problem hiding this comment.
Fixed in 7b94d1c. PgVector.name is now a class attribute and PolarDBPgHNSW overrides it with PolarDBPG, so logs emitted by the base constructor use the correct client name. Added regression coverage for the initialization log.
| pq_nbits = "pq_nbits" | ||
|
|
||
| # PolarDB for PostgreSQL parameters | ||
| hnsw_quantization = "quantization" |
There was a problem hiding this comment.
vectordb_bench/models.py line:177
Low ---- hnsw_quantization = "quantization" duplicates the existing mongodb_quantization_type = "quantization" value, so Python's Enum makes hnsw_quantization a silent alias (CaseConfigParamType.hnsw_quantization.name == "mongodb_quantization_type", and the member is skipped when iterating the enum). The docstring says the value is the key used in CaseConfig.params and the UI, so two unrelated DBs now share the key "quantization". Nothing currently does a reverse lookup, but any future CaseConfigParamType(key) or name-based serialization cannot distinguish them. Consider a distinct value such as "hnsw_quantization".
There was a problem hiding this comment.
Fixed in 7b94d1c. hnsw_quantization now has a distinct enum value. PolarDBPgHNSWConfig accepts both the frontend hnsw_quantization key and the CLI quantization key, preserving index-option generation. Added coverage for enum identity and frontend-to-config mapping.
|
|
||
|
|
||
| def test_polardb_pg_cli_help(): | ||
| result = CliRunner().invoke(PolarDBPgHNSWCommand, ["--help"]) |
There was a problem hiding this comment.
tests/test_polardb_pg.py line:306
Medium ---- The CLI is only exercised via --help, which never builds PolarDBPgConfig/PolarDBPgHNSWConfig or reaches run(); the ~30-option plumbing (PgVectorTypedDict + HNSWFlavor1 + PolarDBPgHNSWOptions -> config constructors -> TaskConfig) is therefore untested. A test invoking the command with required options plus --dry-run would validate that wiring without a live database, since run() returns before benchmark_runner.run() when dry_run is set.
There was a problem hiding this comment.
Fixed in 7b94d1c. Added a real polardbpghnsw --dry-run test with Graph Cache and RaBitQ options. It verifies TaskConfig construction, option propagation, skipped benchmark execution, and password redaction.
| # Search-only benchmarks skip optimize(), so validate and prepare the | ||
| # existing index before any search workers can start. | ||
| if not drop_old and self.case_config.graph_cache: | ||
| with self.init(): |
There was a problem hiding this comment.
vectordb_bench/backend/clients/polardb_pg/polardb_pg.py line:73
Low ---- For search-only runs (drop_old=False) the graph-cache readiness wait runs synchronously inside init during _pre_run, outside the case-level optimize/load timeout wrappers, and is capped only by the hard-coded _GRAPH_CACHE_TIMEOUT_SECONDS = 3600 with no CLI/frontend knob (the tests explicitly assert --graph-cache-timeout is absent). A stuck or slow cache build therefore holds the runner process for up to an hour with no way to tune it. Consider exposing the timeout (or poll interval) as an option so operators can bound search-only runs.
There was a problem hiding this comment.
Fixed in 7b94d1c. Graph Cache wait timeout is now configurable through the Pydantic config, CLI --graph-cache-timeout, and Streamlit UI, with positive-integer validation and a 3600-second default. Lifecycle, CLI, and frontend tests cover the new setting.
Add a pgvector-compatible PolarDB for PostgreSQL HNSW client to the command-line and Streamlit interfaces. Keep it separate from the existing MySQL-compatible PolarDB client. Support PQ, SQ4, SQ8, and RaBitQ index options and optional Graph Cache readiness handling. Reuse the pgvector dependency extra, add configuration and lifecycle tests, and redact connection passwords from PgVector logs.
38b3dbe to
7b94d1c
Compare
3 participants
Summary
This PR adds a pgvector-compatible PolarDB for PostgreSQL HNSW client
to VectorDBBench. It is separate from the existing MySQL-compatible
PolarDB client and does not change the existing
polardb*commands.Supported features
vectordbbench polardbpghnswpgvectordependency extraPolarDB-specific internal quantization and Graph Cache options require
PolarDB vector extension 0.8.3.1 or later.
Implementation
compatible.
the PostgreSQL client implementation.
runs when requested.
Testing
make lintmake lintpip wheel --no-deps .readyandusable=yesbefore search