Sync upstream with gradual SQL view conversion and compatible error tools

README.md

@@ -37,17 +37,19 @@ Some of the key highlights include:
 
 ## Quickstart
 
-Download and run the Parseable binary on your laptop:
+Download the latest published binary from this fork (`VibeEngineeringAI/parseable`).
+The installer requires a published release containing the asset for your platform;
+it will fail if one is unavailable.
 
 ```bash
-curl -fsSL https://logg.ing/install | bash
+curl -fsSL https://raw.githubusercontent.com/VibeEngineeringAI/parseable/main/scripts/download.sh | sh
 ```
 
 <details> 
 <summary>For Windows</summary>
 
 ```pwsh
-powershell -c "irm https://logg.ing/install-windows | iex"
+powershell -c "irm https://raw.githubusercontent.com/VibeEngineeringAI/parseable/main/scripts/download.ps1 | iex"
 ```
 </details>
 

docker-compose-distributed-test-with-kafka.yaml

@@ -72,6 +72,7 @@ services:
       - P_USERNAME=parseableadmin
       - P_PASSWORD=parseableadmin
       - P_CHECK_UPDATE=false
+      - P_SEND_ANONYMOUS_USAGE_DATA=false
       - P_PARQUET_COMPRESSION_ALGO=snappy
       - P_MODE=query
       - RUST_LOG=warn
@@ -113,6 +114,7 @@ services:
       - P_USERNAME=parseableadmin
       - P_PASSWORD=parseableadmin
       - P_CHECK_UPDATE=false
+      - P_SEND_ANONYMOUS_USAGE_DATA=false
       - P_PARQUET_COMPRESSION_ALGO=snappy
       - P_MODE=ingest
       - P_INGESTOR_ENDPOINT=parseable-ingest-one:8000
@@ -161,6 +163,7 @@ services:
       - P_USERNAME=parseableadmin
       - P_PASSWORD=parseableadmin
       - P_CHECK_UPDATE=false
+      - P_SEND_ANONYMOUS_USAGE_DATA=false
       - P_PARQUET_COMPRESSION_ALGO=snappy
       - P_MODE=ingest
       - P_INGESTOR_ENDPOINT=parseable-ingest-two:8000

docker-compose-distributed-test.yaml

@@ -43,6 +43,7 @@ services:
       - P_USERNAME=parseableadmin
       - P_PASSWORD=parseableadmin
       - P_CHECK_UPDATE=false
+      - P_SEND_ANONYMOUS_USAGE_DATA=false
       - P_PARQUET_COMPRESSION_ALGO=snappy
       - P_MODE=query
       - RUST_LOG=warn
@@ -80,6 +81,7 @@ services:
       - P_USERNAME=parseableadmin
       - P_PASSWORD=parseableadmin
       - P_CHECK_UPDATE=false
+      - P_SEND_ANONYMOUS_USAGE_DATA=false
       - P_PARQUET_COMPRESSION_ALGO=snappy
       - P_MODE=ingest
       - P_INGESTOR_ENDPOINT=parseable-ingest-one:8000

docker-compose-gcs-distributed-test.yaml

@@ -18,6 +18,7 @@ services:
       - P_USERNAME=parseableadmin
       - P_PASSWORD=parseableadmin
       - P_CHECK_UPDATE=false
+      - P_SEND_ANONYMOUS_USAGE_DATA=false
       - P_PARQUET_COMPRESSION_ALGO=snappy
       - P_MODE=query
       - RUST_LOG=warn
@@ -33,7 +34,7 @@ services:
       restart_policy:
         condition: on-failure
         delay: 20s
-        max_attempts:
+        max_attempts: 3
     volumes:
       - "${GCS_CREDENTIALS_PATH:-./credentials}:/parseable/svc/:ro,z"
   # ingest server one
@@ -52,6 +53,7 @@ services:
       - P_USERNAME=parseableadmin
       - P_PASSWORD=parseableadmin
       - P_CHECK_UPDATE=false
+      - P_SEND_ANONYMOUS_USAGE_DATA=false
       - P_PARQUET_COMPRESSION_ALGO=snappy
       - P_MODE=ingest
       - P_INGESTOR_ENDPOINT=parseable-ingest-one:8000

docker-compose-test-with-kafka.yaml

@@ -44,6 +44,7 @@ services:
       - P_USERNAME=parseableadmin
       - P_PASSWORD=parseableadmin
       - P_CHECK_UPDATE=false
+      - P_SEND_ANONYMOUS_USAGE_DATA=false
       - P_PARQUET_COMPRESSION_ALGO=snappy
       - P_KAFKA_CONSUMER_TOPICS=test-logs-stream
       - P_KAFKA_BOOTSTRAP_SERVERS=kafka-0:9092

docker-compose-test.yaml

@@ -42,6 +42,7 @@ services:
       - P_USERNAME=parseableadmin
       - P_PASSWORD=parseableadmin
       - P_CHECK_UPDATE=false
+      - P_SEND_ANONYMOUS_USAGE_DATA=false
       - P_PARQUET_COMPRESSION_ALGO=snappy
       - MAX_PERIODIC_LOCAL_SYNC_CYCLES=1
       - RUST_LOG=warn

docs/sql-filter-conversion.md

@@ -0,0 +1,79 @@
+# Saved SQL filter compatibility and conversion
+
+Dataset-scoped SQL filters and upstream SQL filters using `stream_name: "sql"`
+are both supported. Existing views are not automatically converted. New native
+Logs views keep their dataset-scoped format until the owner chooses **Convert to
+SQL view** in Saved views. Converted views still open in the original dataset;
+`explorer_state` retains their dataset, query controls, time range and display
+settings. Editing a converted view keeps it converted.
+
+The native explorer reads both formats. Classic filters without native explorer
+state remain visible under their original dataset, but cannot be opened or
+converted by the native explorer. The literal `sql` is a reserved dataset name,
+so it cannot collide with a user-created dataset.
+
+## Backend operation
+
+```
+POST /api/v1/filters/{filter_id}/convert-to-sql
+```
+
+The endpoint takes no JSON payload and returns the complete saved filter. It
+requires filter-update permission and an owner-or-admin check. Only SQL filters
+can be converted. The server copies the authorized stored record, preserving
+its ID, owner, query, time filter and extra metadata. Repeating the request is
+safe, including after an interrupted cleanup. Admin updates also preserve the
+original owner.
+
+Ordinary create and update requests accept a valid dataset name or, for SQL
+filters only, the literal `sql`. Path-like names remain invalid. A stale client
+cannot turn a converted SQL filter back into a dataset-scoped one by sending its
+old stream name.
+
+## Reconciliation and recovery
+
+Persistence uses `users/<owner>/filters/<stream>/<id>.json`. Conversion writes
+the `sql` copy first, updates the server cache, then removes obsolete copies of
+the same owner's filter ID. Cleanup failure returns an error, while the durable
+new copy remains available for retry. Deletion removes aliases before the
+current copy, so a successful deletion cannot leave an older copy to return on
+restart.
+
+Startup reconciliation groups records by owner and ID within each tenant. When
+both canonical SQL and legacy SQL copies exist, it deterministically keeps the
+canonical copy and removes obsolete SQL copies. A legacy record with no
+canonical copy is left in its existing format. Ambiguous legacy-only duplicates
+are selected deterministically without deleting the alternatives. Cleanup
+failures are logged and can be retried on the next startup or conversion; there
+is no one-time completion flag that could hide unfinished work.
+
+Older administrator edits could also leave the same filter ID under different
+owners. These records are retained independently. Each owner, including an
+administrator, can manage their own copy; an administrator without a copy can
+manage another owner's record only when the ID is unambiguous. The native
+Saved views selector distinguishes the owners' copies.
+
+A process-wide async lock serializes filter persistence and cache changes.
+This follows the existing single-query-server ownership model: it is not a
+distributed lock, and concurrent writes through multiple independent query
+servers sharing the same metastore are not coordinated by this change.
+
+Before a rollout, back up stored filter metadata and verify a legacy view can
+be opened, converted, edited, restarted and deleted without reappearing. A
+rollback to an older native UI will hide converted views because that UI only
+understands dataset-scoped records; keep this compatible UI when rolling back
+other changes. Conversion does not change log data or delete datasets.
+
+## Regression checks
+
+Run the isolated HTTP/restart smoke test against a newly built server:
+
+```sh
+PARSEABLE_BIN=/absolute/path/to/parseable python3 scripts/test-sql-filter-conversion.py
+```
+
+It starts its own local-store process and temporary data directories; it does not
+contact an existing deployment. The fixture checks owner authorization, admin
+updates, conversion retries, stale-client edits, startup cleanup of an injected
+legacy duplicate, and delete followed by restart. Rust metastore tests also
+exercise cleanup failures, tenant isolation, and ambiguous legacy records.

docs/tool-api.md

@@ -21,10 +21,13 @@ Response:
     {
       "name": "get_dataset_schema",
       "title": "Get dataset schema",
-      "description": "Get columns and data types for a dataset.",
+      "description": "Get the full Arrow JSON schema by default, or request format=compact for tab-separated column names and types.",
       "inputSchema": {
         "type": "object",
-        "properties": { "dataset": { "type": "string", "minLength": 1 } },
+        "properties": {
+          "dataset": { "type": "string", "minLength": 1 },
+          "format": { "type": "string", "enum": ["json", "compact"], "default": "json" }
+        },
         "required": ["dataset"],
         "additionalProperties": false
       }
@@ -89,3 +92,38 @@ tool.
 The REST API is not itself an MCP transport. An MCP adapter maps the registry
 response to `tools/list`, forwards calls to this endpoint, and returns the
 `CallToolResult` from Parseable through MCP `tools/call`.
+
+## Schema output compatibility
+
+`get_dataset_schema` preserves the full Arrow schema object in
+`structuredContent.result` when `format` is omitted or `"json"`. Existing JSON
+consumers do not need to change. Set `"format": "compact"` to receive a string
+with a `name\ttype` header and one tab-separated row per field. Compact output
+escapes tabs, newlines, carriage returns and backslashes and omits metadata and
+top-level nullability; it is for inspection, not Arrow schema interchange.
+The HTTP schema endpoint likewise defaults to JSON and accepts `?format=compact`.
+
+## Error tools
+
+`list_log_errors` and `list_trace_errors` each require `dataset`, `startTime`,
+and `endTime`; `limit` defaults to 50 and is bounded to 1–500. They enforce query
+permissions and require logs and traces telemetry, respectively.
+The log tool applies exact, half-open time bounds (start inclusive, end exclusive)
+to the dataset's configured time partition, or `p_timestamp` by default; `lastSeen`
+uses the same timestamp column. Subminute ranges retain their full precision.
+
+`list_log_errors` classifies records by explicit severity, following the native
+Errors page's policy. It checks every available `level`, `log_level`, `severity`,
+`severity_text` and `severity_description` column using case-insensitive, trimmed
+severity names, plus OTel `severity_number` codes 17–24. Pino/Bunyan codes 50 and
+60 are accepted only in `level`. Invalid numeric text and fractional codes do
+not fail the query or count as valid severities. A dataset without a supported
+severity column produces an actionable tool error; message substrings,
+`p_log_category` and the presence of exception metadata do not independently
+classify records as errors.
+
+The log tool groups by service and error type, returns a sample message, count
+and last-seen value, and coalesces supported aliases per row. Its `total` is the
+number of returned groups, not an unbounded count of all matching groups. The
+native Errors page additionally separates issues by message, so its grouping
+and pagination are not identical to this summary tool.

frontend/README.md

@@ -74,7 +74,7 @@ This creates `build.zip` with a top-level `dist/` directory, including `dist/ind
 - Keystone and LLM settings: native capability pages with explicit limitations and deployment-dependent classic links. See [native screen scope](docs/prism-parity.md#native-missing-screen-scope-2026-10-12).
 - Component gallery at `/components`, plus isolated Storybook stories for UI, PromQL, charts, Traces, APM and ingestion setup.
 
-The default log query reads at most 100 rows, with choices up to 10,000. Message search assumes a `message` field. The histogram separately aggregates every matching event through `/query`, using the same captured bounds and predicates as the rows. Its 48 buckets divide elapsed time evenly; they are not calendar-day buckets. AND/OR filters have two levels, at most 12 groups and 50 rules, and support `=`/`!=`; SQL NULL values are excluded by both operators. Saved log views use server `/filters` CRUD, while copy links encode versioned explorer state with fixed absolute bounds. Context uses the existing `/query/context` endpoint with a 101-event bound and requires `p_timestamp` plus a string `message`, `body` or `log`. Large/wide results virtualize rows and columns; sorting and export cover loaded rows only. Wrapped virtual rows show a 96px preview; event details retain the full value. Timezones change display and wall-clock editing while requests retain UTC instants; DST gaps are rejected and repeated times require an earlier/later choice. Demo SQL intentionally supports only a small SELECT subset and rejects unsupported syntax. Full SQL requires a server. SQL multi-dataset generation uses `UNION ALL BY NAME`, fills missing columns with NULL and requires compatible types for common columns, and keeps the 100-row default; demo does not support unions. Incoming query links open or focus a tab while preserving other drafts. Reopening an already-open saved query focuses that tab and preserves its edits. The saved SQL library uses persistent browser local storage, while draft workspaces use session storage independent to each browser tab and survive navigation/reload. Both are scoped to the server and signed-in user (with a separate demo scope); they are not server-synced, and result rows are not persisted. Query handoff parameters are consumed after opening the tab so reloading an edited draft does not reopen the original query. SQL chart previews use the executed query and captured time bounds. Adding a chart to an existing dashboard preserves that dashboard’s time range; a new dashboard starts with the executed range.
+The default log query reads at most 100 rows, with choices up to 10,000. Message search assumes a `message` field. The histogram separately aggregates every matching event through `/query`, using the same captured bounds and predicates as the rows. Its 48 buckets divide elapsed time evenly; they are not calendar-day buckets. AND/OR filters have two levels, at most 12 groups and 50 rules, and support `=`/`!=`; SQL NULL values are excluded by both operators. Saved log views use server `/filters` CRUD and support dataset-scoped and canonical SQL views. New saves stay dataset scoped; owners can explicitly convert compatible views to SQL while retaining their query and dataset. Copy links encode versioned explorer state with fixed absolute bounds. Context uses the existing `/query/context` endpoint with a 101-event bound and requires `p_timestamp` plus a string `message`, `body` or `log`. Large/wide results virtualize rows and columns; sorting and export cover loaded rows only. Wrapped virtual rows show a 96px preview; event details retain the full value. Timezones change display and wall-clock editing while requests retain UTC instants; DST gaps are rejected and repeated times require an earlier/later choice. Demo SQL intentionally supports only a small SELECT subset and rejects unsupported syntax. Full SQL requires a server. SQL multi-dataset generation uses `UNION ALL BY NAME`, fills missing columns with NULL and requires compatible types for common columns, and keeps the 100-row default; demo does not support unions. Incoming query links open or focus a tab while preserving other drafts. Reopening an already-open saved query focuses that tab and preserves its edits. The saved SQL library uses persistent browser local storage, while draft workspaces use session storage independent to each browser tab and survive navigation/reload. Both are scoped to the server and signed-in user (with a separate demo scope); they are not server-synced, and result rows are not persisted. Query handoff parameters are consumed after opening the tab so reloading an edited draft does not reopen the original query. SQL chart previews use the executed query and captured time bounds. Adding a chart to an existing dashboard preserves that dashboard’s time range; a new dashboard starts with the executed range.
 
 Logs implementation keeps persisted settings in `logViewReducer.ts`, with explicit dataset reset, saved-view and classic-link transitions. `logFilters.ts` owns expression limits and additions; adding within a group and narrowing the whole expression remain distinct operations. `logQuery.ts` compiles canonical criteria once for the page's row and histogram queries. Changes to client-only filter IDs or empty groups do not refresh the captured time window. Histogram responses carry their bucket geometry to the renderer. Both table modes use `logTable.tsx` for sorting, labels and cells. `logsContract.ts` validates server payloads; `demoLogs.ts` implements the demo methods separately. Saved-view instants, server timestamps and timezone wall-clock inputs retain separate validators because their accepted formats differ.