diff --git a/frontend/README.md b/frontend/README.md index df555fd72..df2909ffa 100644 --- a/frontend/README.md +++ b/frontend/README.md @@ -67,7 +67,8 @@ This creates `build.zip` with a top-level `dist/` directory, including `dist/ind - Dashboards: server-backed lists and detail routes, search/tags/favourites/sort/pagination, ownership, full-document saves with conflict resolution, SQL and PromQL tiles, six variable types, URL time ranges and selections, classic import/export, and explicit migration of old browser-local dashboards. Demo dashboards use independent in-memory state per client. - Team: native users and one-time passwords, role and privilege management, default OIDC role configuration, provider group mappings and role provenance, and API key creation/copy/deletion. Search and 25-row pagination apply to every tab; authorization stays with the backend. - Agents: schema-discovered GenAI telemetry datasets, bounded recent records, time ranges, local search, record inspection and trace handoff. -- Cluster: registered node reachability and resource metrics with independent request states; read-only. Errors: paginated OTLP error spans and trace handoff; classic grouped issue analytics remain unavailable. +- Cluster: registered node reachability and resource metrics with independent request states; read-only. +- Errors: grouped issues, full-range frequency analytics, and occurrence inspection for OTLP traces and recognized log severity schemas, plus paginated OTLP error spans and trace handoff. Groups match exact service, error type and message values; issue assignment and resolution are not persisted. - Workspace setup: `/onboarding` discovers accessible datasets and opens the matching explorer; `/new` explains server scope and provides connection/setup actions. Hosted workspace creation is not exposed by this server. - Network access: edit the server’s outbound alert-target policy, including domain/CIDR rules, private IP access and TLS verification. - Keystone and LLM settings: native capability pages with explicit limitations and deployment-dependent classic links. See [native screen scope](docs/prism-parity.md#native-missing-screen-scope-2026-10-12). diff --git a/frontend/docs/api-contracts.md b/frontend/docs/api-contracts.md index 17a92c17f..1f37228d6 100644 --- a/frontend/docs/api-contracts.md +++ b/frontend/docs/api-contracts.md @@ -213,7 +213,11 @@ Agents uses existing dataset list, schema, metadata and SQL requests. Discovery Cluster reads `GET /api/v1/cluster/info` and `/api/v1/cluster/metrics` through the shared authenticated client. The first returns node identity, storage/staging paths, reachability and nullable status/error text. Metrics are independent: a denied metrics request does not hide node health. Nonfinite Rust metric values serialized as null display as unavailable per field. Missing cluster routes (404/405) show a deployment-mode explanation; 403 and 401 retain shared permission/session handling. No node deletion is exposed. -Errors uses the existing trace-list contract with `options: "errorSpans"`, selectable UTC ranges, server sorting and pagination. The selected dataset must advertise `otel-traces`. Failed metadata inspection is distinguished from confirmed incompatibility and can be retried. Dataset/range changes reset pagination and cancel obsolete requests. Trace handoff preserves the captured query bounds. Classic grouped-issue and frequency endpoints are absent and are not called. +Errors aggregates grouped issues and frequency through the existing `POST /api/v1/query` contract. Dataset metadata and schema determine the trace or log interpretation and the time partition. OTLP spans are deduplicated before aggregation; log occurrences count severity-matched records. Exact service, error type and message values identify a group, with SQL NULL kept distinct from literal display placeholders. Aggregates cover the full captured interval independently of paginated issue and occurrence lists. SQL enforces precise half-open bounds inside a minute-expanded transport envelope. + +Trace type and message come from one deterministic representative row, preferring populated exception events, so different events cannot form an invented pair. Logs recognize textual severity aliases, exact OTLP severity codes 17–24 and Pino/Bunyan `level` codes 50/60; `msg` is a recognized message alias. The log SQL handoff returns full stored records, with internal query metadata retained only in sample mode. See [the Errors runbook](error-analytics.md) for alias behavior, counting rules and real-server tests. + +The Error spans view retains the native trace-list contract with `options: "errorSpans"`, server sorting and pagination for datasets advertising `otel-traces`. Failed metadata or schema requests remain explicit and retryable. Dataset/range changes cancel obsolete requests and reset pagination. Trace and SQL investigation links preserve captured query bounds. No classic grouped-issue endpoint or persisted issue lifecycle API is required. Network access reads and replaces `GET/PUT /api/v1/alert-target-policy`. The complete object contains `allowPrivate`, `allowInvalidTls`, `allowedDomains`, `allowedCidrs`, `deniedDomains`, and `deniedCidrs`. All fields are required; the server is authoritative for authorization, CIDR validation and allow/deny overlaps. This policy governs outbound alert delivery, not ingress access to Parseable. The editor retains failed-save drafts and protects unsaved navigation. Demo writes are isolated within each client. The backend caches policy per node and loads it on startup; cross-node propagation is not guaranteed by a successful save. diff --git a/frontend/docs/error-analytics.md b/frontend/docs/error-analytics.md new file mode 100644 index 000000000..72a6493a4 --- /dev/null +++ b/frontend/docs/error-analytics.md @@ -0,0 +1,53 @@ +# Errors: grouping, frequency, and regression tests + +The Errors page queries the existing SQL API for grouped issues and frequency. It also retains the native OTLP error-span list. The [Mayfly implementation record](https://mayfly-celld.tail835e0d.ts.net/w/46ncdGypknzAj7dnRkJ8bQ?page=1983eae1-fba9-4616-9643-1de4e746162c) records the scope and validation evidence. + +## What the counts mean + +An issue is an exact match of service, error type, and message. Empty strings and missing values are treated as absent; absent values remain distinct from literal text such as `(unknown)`. Variable IDs or numbers in messages create separate issues. There is no fuzzy grouping, saved resolution state, assignment, or cross-dataset correlation. + +For OTLP traces, an occurrence is a distinct error span. The query deduplicates expanded event/link rows and retries by service, trace ID, and span ID before aggregating. A span must have error status `2`; an exception event alone does not make a successful span an error. When rows disagree, type and message come together from one representative row. A populated exception event takes priority over direct span attributes; ties use type then message in ascending order, with absent values last. This keeps groups deterministic without inventing a type/message combination that never occurred. The occurrence timestamp remains the earliest row in the selected interval. + +For logs, an occurrence is one stored record. Repeated identical logs remain separate occurrences. Recognized textual severity fields are checked case-insensitively for error/fatal equivalents. OTLP numeric severity must be an integer code from 17 through 24; numeric `level` supports the standard Pino/Bunyan error (50) and fatal (60) codes. Fractional codes do not match. Message text is not searched for the word “error” to classify a record. The derived `p_log_category` field is excluded because OTLP ingestion can infer it from body text when severity is unspecified. A dataset with only that category and no explicit severity field is unsupported. Schema inspection selects available service, type and message aliases; unsupported schemas have an explicit explanation and a Log explorer link. + +Each view covers one dataset and one captured time interval. The dataset's configured time partition defines that interval. SQL applies `start <= timestamp < end` precisely, while the request envelope expands to minute boundaries for the server planner. Frequency buckets divide the selected interval into at most 48 equal elapsed-time intervals, including zero-count buckets. Summary totals and frequency are independent of the current issue page. + +Issue and occurrence pages request one extra row to decide whether another page exists. Sorting happens on the server with deterministic group tie-breakers. Dataset/range changes and refresh discard obsolete requests and selections. Native trace links preserve the captured bounds. Log investigation opens SQL with the same classification and exact issue predicates, returning full stored records so stack traces and other fields remain available. Live handoffs start with readable SQL; only sample mode retains the internal query marker needed by its restricted evaluator. A log trace ID does not imply a compatible trace dataset. + +## Code map + +- `src/lib/errorAnalytics.ts`: schema interpretation, safely quoted SQL, strict result decoding, loaders and group identity. +- `src/lib/errorAnalyticsDemo.ts`: deterministic sample-data evaluation, restricted to exact queries rebuilt from their metadata. +- `src/features/errors/`: page orchestration, grouped issues, frequency and occurrence inspection. +- `src/lib/errorAnalytics.test.ts`: fast data-contract and sample-data regressions. +- `src/lib/errorAnalytics.live.test.ts`: opt-in generated SQL checks against a real Parseable server. +- `e2e/errors.spec.ts`: browser state, navigation, pagination and failure behavior. API responses use fixture records; these tests do not establish SQL engine compatibility. + +## Running the tests + +From `frontend`, install dependencies with `npm ci`, then run focused unit tests: + +```sh +npx vitest run src/lib/errorAnalytics.test.ts +``` + +The real-server suite is deliberately opt-in. It creates uniquely named disposable JSON-log, OTLP-log and OTLP-trace datasets and removes those datasets afterward. Use an isolated development server and an account allowed to ingest, query, and delete these fixtures: + +```sh +ERRORS_REFERENCE_URL=http://127.0.0.1:8450 \ +ERRORS_REFERENCE_USERNAME=fixture \ +ERRORS_REFERENCE_PASSWORD=fixture-throwaway \ +npx vitest run src/lib/errorAnalytics.live.test.ts +``` + +The example credentials belong to the local disposable fixture server. Without `ERRORS_REFERENCE_URL`, the suite skips its live tests. It derives bounds from persisted timestamps and polls for readiness instead of assuming ingestion is immediately queryable. Assertions use explicit fixture counts: they check retries and event expansion, nonerror exclusion, severity aliases, escaped message values, NULL identity, more than one page of groups, and agreement between frequency and summary counts. + +Browser tests use the repository's Playwright configuration: + +```sh +npx playwright test e2e/errors.spec.ts e2e/operations.spec.ts --workers=1 +``` + +Ensure the configured server serves this checkout. A pre-existing server on the default port can belong to another worktree. Production build/type checking and the complete frontend unit suite are `npm run build` and `npm test`. + +The Errors stories cover trace and log issues, loading, empty, permission, query failure, unsupported schema and unavailable metadata states. After building Storybook, run `native-screens.spec.ts --grep operations-errors` with `playwright.storybook.config.ts`. Detail stories exercise paging, trace links, keyboard dismissal and focus restoration; the suite also runs axe and a narrow dark viewport. Follow the shared Playwright lock instructions in the frontend README on shared hosts. diff --git a/frontend/docs/prism-parity.md b/frontend/docs/prism-parity.md index bbdaa2b6a..f97bcdf0a 100644 --- a/frontend/docs/prism-parity.md +++ b/frontend/docs/prism-parity.md @@ -126,15 +126,15 @@ The [observability validation record](observability-validation.md) documents ori The previously classic-only routes below now resolve to native pages. `classicUi.ts` marks them `limited`, distinguishing a usable native subset or capability explanation from full classic parity. The existing Traces, APM, Add data and telemetry-aware dataset routes retain their behavior. -| Routes | Native behavior | Deliberate limits | -| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------ | -| `/agents`, `/agents/:dataset` | Discover flattened GenAI fields through dataset schemas; query the latest 100 telemetry records in a selected time range; search loaded records; inspect full records; open compatible trace IDs | Individual events/spans, not classic run aggregation, evaluations, pricing or token-cost analytics. Demo uses explicit synthetic fixtures. | -| `/cluster` | Show registered distributed nodes, reachability, status, storage/staging locations and resource metrics from the existing cluster endpoints | Read-only. Standalone servers can report cluster administration unavailable. Node and metrics failures remain separate. | -| `/errors`, `/errors/:dataset` | Discover accessible OTLP trace datasets; list, sort and paginate error spans; open their trace waterfall | Classic grouped error issues and frequency endpoints are absent from this backend; arbitrary log issue aggregation is not fabricated. | -| `/onboarding` | Discover accessible datasets, link each to its confirmed explorer, and guide ingestion and team setup | Dataset presence is not presented as proof of event arrival. Existing Add data owns ingestion configuration and verification. | -| `/new` | Explain server workspace scope; open connection settings or native setup | No API in this checkout provisions a separate hosted workspace. | -| `/keystone`, `/keystone/:id` | Native availability page and classic investigation handoff where available | No supported investigation/conversation API in this checkout. No claim that an ID exists or is accessible. | -| `/settings/llm-key` | Native availability pages and deployment-dependent classic handoff | No provider-key management API. No credentials are read, stored or submitted. | +| Routes | Native behavior | Deliberate limits | +| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `/agents`, `/agents/:dataset` | Discover flattened GenAI fields through dataset schemas; query the latest 100 telemetry records in a selected time range; search loaded records; inspect full records; open compatible trace IDs | Individual events/spans, not classic run aggregation, evaluations, pricing or token-cost analytics. Demo uses explicit synthetic fixtures. | +| `/cluster` | Show registered distributed nodes, reachability, status, storage/staging locations and resource metrics from the existing cluster endpoints | Read-only. Standalone servers can report cluster administration unavailable. Node and metrics failures remain separate. | +| `/errors`, `/errors/:dataset` | Discover trace and log error datasets; group issues, query full-range frequency, inspect occurrences, sort/page issues and OTLP spans, and open trace or SQL investigations | Exact service/type/message grouping with recognized severity schemas; persistent issue resolution, fuzzy grouping and cross-dataset correlation remain unavailable. See [Errors contracts and tests](error-analytics.md). | +| `/onboarding` | Discover accessible datasets, link each to its confirmed explorer, and guide ingestion and team setup | Dataset presence is not presented as proof of event arrival. Existing Add data owns ingestion configuration and verification. | +| `/new` | Explain server workspace scope; open connection settings or native setup | No API in this checkout provisions a separate hosted workspace. | +| `/keystone`, `/keystone/:id` | Native availability page and classic investigation handoff where available | No supported investigation/conversation API in this checkout. No claim that an ID exists or is accessible. | +| `/settings/llm-key` | Native availability pages and deployment-dependent classic handoff | No provider-key management API. No credentials are read, stored or submitted. | Network access (`/settings/network-access`) edits the real outbound alert-target policy via `GET` and `PUT /api/v1/alert-target-policy`. It is distinct from ingress access control. The full policy includes private-IP and invalid-TLS switches plus allowed/denied domains and CIDRs. Backend authorization and validation remain authoritative; failed saves retain the draft. The policy cache is per server node and loaded at startup; cross-node propagation is a backend limitation, not asserted by the UI. diff --git a/frontend/docs/validation-review.md b/frontend/docs/validation-review.md index 35176c918..416f76ba4 100644 --- a/frontend/docs/validation-review.md +++ b/frontend/docs/validation-review.md @@ -655,3 +655,22 @@ Validation of the final implementation: - Opus findings addressed include network API scope, full-policy payloads, domain rules that could never match the backend, authorization-header allowlist help, permission-safe discard, unique field descriptions, missing metric values, failed dataset verification, unnecessary discovery requests, duplicate refresh requests and native URL preservation. An independent regression caught and fixed synthetic demo data appearing under unknown dataset names. Broader suites are **not clean** on this host. The 493-test development-server run was stopped after repeated `net::ERR_INSUFFICIENT_RESOURCES`, blank lazy pages and Chromium failures: 238 passed, 45 failed, 3 interrupted, 1 skipped and 206 not run. The full 221-test Storybook run had 162 passes and 59 failures (38 existing screenshot assertions and 21 runtime/resource failures). A representative grouped-bars screenshot was rebuilt and tested from an isolated `origin/main` archive: it fails by the same 1,093 pixels, and its actual PNG is byte-identical to this branch (`c21e1b584291ac5ee37e020c75832fb14dbec1566610613dc06c792f9249b06d`). No baselines were updated. The obsolete sidebar expectation was updated for Agents/Errors and passed on retry; the reported APM 24-hour test also passed independently. The final focused run used a production bundle to avoid the development server's large module graph. + +## Grouped Errors analytics (2026-10-12) + +Implemented on `feat/error-issue-analytics` from `origin/main` (`bd882956`) in an isolated worktree. Two Sol 6.1 agents covered wiki research and real-server validation; Opus 5.5 agents covered architecture, SQL/demo implementation, UI, and independent review. The [implementation runbook](error-analytics.md) explains grouping, counting and reproducible test commands. The [Mayfly record](https://mayfly-celld.tail835e0d.ts.net/w/46ncdGypknzAj7dnRkJ8bQ?page=1983eae1-fba9-4616-9643-1de4e746162c) records completion evidence. + +The default Errors view now groups exact service/type/message values, shows full-range counts and frequency, and opens paginated occurrences. General log severity schemas and OTLP traces are supported. The original paginated, sortable error-span view and trace links remain available. Live SQL investigation returns complete stored log records with exact issue predicates; the same handoff also executes in sample mode. + +Review regressions cover coherent exception pairs when a span has multiple different exception events, numeric Pino/Bunyan levels, exact numeric status/severity comparisons, direct versus event exception fields, permission versus unsupported-schema states, and readable live SQL handoffs. Frequency and summary counts remain independent of table pagination. No fuzzy grouping or persisted assignment/resolution state is claimed. + +The final independent Opus review confirmed the earlier findings resolved and reported no required fixes. Optional follow-ups include measuring large-range trace window-query cost and improving errors for hand-edited malformed demo markers. + +Validation passed: + +- Production build and TypeScript; all **2,193 frontend unit tests**, with ten opt-in live tests skipped in that default run. +- **27 Errors unit tests and nine real-server tests** against the disposable Parseable server at `127.0.0.1:8450`. Fixtures cover JSON/OTLP logs and traces, severity aliases and numeric levels, exact grouping, differing exception events, retries, SQL quoting, raw log handoff, pagination, fractional bounds, and frequency totals. Created datasets are removed afterward. +- **12 application browser tests** (`errors.spec.ts` and `operations.spec.ts`) and **12 Errors Storybook checks**, including axe, occurrence paging, keyboard dismissal/focus restoration, and a narrow dark viewport. Final browser runs held the shared Playwright lock and used this worktree's isolated ports 5191/6191 instead of other checkouts' servers on the defaults. +- Formatting and `git diff --check`. + +The browser transport fixtures establish UI behavior; the opt-in live suite separately establishes generated SQL compatibility. Large-production-dataset performance, cross-browser coverage, and deployment are outside this validation. The full unrelated browser and screenshot suites were not rerun; the earlier baseline limitations above still apply. diff --git a/frontend/e2e/errors.spec.ts b/frontend/e2e/errors.spec.ts new file mode 100644 index 000000000..4d3ed8c12 --- /dev/null +++ b/frontend/e2e/errors.spec.ts @@ -0,0 +1,278 @@ +import { expect, test, type Page } from '@playwright/test'; +import { handleErrorDemoQuery } from '../src/lib/errorAnalyticsDemo'; +import type { LogRecord, QueryRequest } from '../src/lib/types'; + +// Browser tests exercise request/state/navigation behavior. The separate +// errorAnalytics.live.test.ts executes the generated SQL on a real server. +const bounds = { + startTime: '2026-10-12T15:00:00.123Z', + endTime: '2026-10-12T16:00:00.123Z', +}; +const range = new URLSearchParams({ start: bounds.startTime, end: bounds.endTime }); +const logRows: LogRecord[] = [ + // Identical logs are separate occurrences: six retries form one issue. + ...Array.from({ length: 6 }, (_, index) => ({ + p_timestamp: `2026-10-12T15:0${index}:30.000Z`, + level: 'ERROR', + service: 'checkout', + 'exception.type': 'TimeoutError', + message: "Payment gateway's connection timed out", + })), + // More groups than one page prove that the overview is not a page subtotal. + ...Array.from({ length: 31 }, (_, index) => ({ + p_timestamp: '2026-10-12T15:20:00.000Z', + level: 'ERROR', + service: 'worker', + 'exception.type': 'JobError', + message: `Job failure ${String(index).padStart(2, '0')}`, + })), + { + p_timestamp: '2026-10-12T15:30:00.000Z', + level: 'INFO', + service: 'checkout', + message: 'No error occurred', + }, +]; + +const traceBase = { + p_timestamp: '2026-10-12T15:10:00.000Z', + 'service.name': 'checkout', + span_trace_id: 'a'.repeat(32), + span_span_id: '1'.repeat(16), + span_name: 'charge-card', + span_status_code: 2, + event_name: 'exception', + 'event_exception.type': 'TimeoutError', + 'event_exception.message': 'Payment gateway timeout', +}; +const traceRows: LogRecord[] = [ + traceBase, + { ...traceBase }, // Expanded event/retry: still just one span. + { ...traceBase, span_trace_id: 'b'.repeat(32), span_span_id: '2'.repeat(16) }, + { ...traceBase, span_span_id: '3'.repeat(16), span_status_code: 1 }, // Exception alone is insufficient. +]; + +type CapturedQuery = QueryRequest & { + kind: string; + offset?: number; + sort?: string; + issue?: unknown; +}; + +async function serveLogs( + page: Page, + options: { unsupported?: boolean; denied?: boolean; traces?: boolean } = {}, +) { + const requests: CapturedQuery[] = []; + let denied = options.denied ?? false; + await page.route('**/api/**', async (route) => { + const path = new URL(route.request().url()).pathname; + if (path === '/api/v1/logstream') + return route.fulfill({ + json: [{ name: options.traces ? 'orders_traces' : 'application_logs' }], + }); + if (path.endsWith('/info')) + return route.fulfill({ + json: { + telemetryType: options.traces ? 'traces' : 'logs', + logSource: [{ log_source_format: options.traces ? 'otel-traces' : 'json' }], + }, + }); + if (path.endsWith('/schema')) + return route.fulfill({ + json: { + fields: (options.traces + ? Object.keys(traceBase) + : options.unsupported + ? ['p_timestamp', 'message'] + : ['p_timestamp', 'level', 'service', 'exception.type', 'message'] + ).map((name) => ({ name, data_type: 'Utf8' })), + }, + }); + if (path === '/api/v1/query') { + const body = route.request().postDataJSON(); + const request: QueryRequest = { + sql: body.query, + startTime: body.startTime, + endTime: body.endTime, + }; + const match = /^\/\* parseable-errors:([^\n]+) \*\//.exec(request.sql); + if (match) { + requests.push({ ...request, ...JSON.parse(decodeURIComponent(match[1])) }); + if (denied) return route.fulfill({ status: 403, body: 'Error analytics access denied' }); + return route.fulfill({ + json: handleErrorDemoQuery(request, options.traces ? traceRows : logRows), + }); + } + return route.fulfill({ json: [] }); + } + return route.fulfill({ json: {} }); + }); + return { requests, allowQueries: () => (denied = false) }; +} + +test('log issues show full-range totals while paging and sorting groups on the server', async ({ + page, +}) => { + const fixture = await serveLogs(page); + await page.goto(`/errors/application_logs?${range}`); + await expect(page.getByRole('heading', { name: 'Errors', exact: true })).toBeVisible(); + await expect( + page.getByText("Payment gateway's connection timed out", { exact: true }), + ).toBeVisible(); + await expect(page.getByRole('img', { name: /Error log records over time/ })).toBeVisible(); + await expect( + page + .locator('dt') + .filter({ hasText: /^Error log records$/ }) + .locator('+ dd'), + ).toHaveText('37'); + await expect( + page + .locator('dt') + .filter({ hasText: /^Issues$/ }) + .locator('+ dd'), + ).toHaveText('32'); + await page.getByRole('button', { name: 'Next page', exact: true }).click(); + await expect + .poll(() => + fixture.requests.some((request) => request.kind === 'issues' && request.offset === 25), + ) + .toBe(true); + await expect( + page + .locator('dt') + .filter({ hasText: /^Error log records$/ }) + .locator('+ dd'), + ).toHaveText('37'); + await page.getByRole('button', { name: 'Sort by Last seen (UTC)', exact: true }).click(); + await expect + .poll(() => + fixture.requests.some( + (request) => + request.kind === 'issues' && request.sort === 'latest' && (request.offset ?? 0) === 0, + ), + ) + .toBe(true); +}); + +test('a log issue opens its matching occurrences and hands the exact query to SQL', async ({ + page, +}) => { + const fixture = await serveLogs(page); + await page.goto(`/errors/application_logs?${range}`); + await page.getByRole('button', { name: /Payment gateway's connection timed out/ }).click(); + const detail = page.getByRole('dialog'); + await expect(detail).toBeVisible(); + await expect + .poll(() => fixture.requests.some((request) => request.kind === 'occurrences')) + .toBe(true); + const link = detail.getByRole('link', { name: /SQL/ }); + const href = new URL((await link.getAttribute('href'))!, 'http://localhost'); + expect(href.pathname).toBe('/sql-editor'); + expect(href.searchParams.get('start')).toBe('2026-10-12T15:00:00.000Z'); + expect(href.searchParams.get('end')).toBe('2026-10-12T16:01:00.000Z'); + expect(href.searchParams.get('query')).toContain(bounds.startTime); + expect(href.searchParams.get('query')).toContain(bounds.endTime); + expect(href.searchParams.get('query')).toContain("Payment gateway''s connection timed out"); + expect(href.searchParams.get('query')).toMatch(/^SELECT \* FROM /); + expect(href.searchParams.get('query')).not.toContain('parseable-errors:'); + // A trace id in a log is not evidence of a compatible trace dataset. + await expect(detail.getByRole('link', { name: /Open trace/ })).toHaveCount(0); + await page.keyboard.press('Escape'); + await expect(detail).toHaveCount(0); +}); + +test('permission failures stay errors and retry restores real analytics', async ({ page }) => { + const fixture = await serveLogs(page, { denied: true }); + await page.goto(`/errors/application_logs?${range}`); + await expect(page.getByText('Permission denied', { exact: true }).first()).toBeVisible(); + await expect(page.getByText('No errors', { exact: true })).toHaveCount(0); + fixture.allowQueries(); + await page + .getByRole('region', { name: 'Issues', exact: true }) + .getByRole('button', { name: 'Try again', exact: true }) + .click(); + await expect( + page.getByText("Payment gateway's connection timed out", { exact: true }), + ).toBeVisible(); +}); + +test('logs without a recognized severity field explain eligibility without guessing errors from text', async ({ + page, +}) => { + const fixture = await serveLogs(page, { unsupported: true }); + await page.goto(`/errors/application_logs?${range}`); + await expect(page.getByRole('link', { name: 'Explore logs' })).toHaveAttribute( + 'href', + '/logs/explore/application_logs', + ); + expect(fixture.requests).toHaveLength(0); +}); + +test('trace issue drill-down preserves exact bounds and links each deduplicated occurrence', async ({ + page, +}) => { + await serveLogs(page, { traces: true }); + await page.goto(`/errors/orders_traces?${range}`); + await expect( + page + .locator('dt') + .filter({ hasText: /^Error spans$/ }) + .locator('+ dd'), + ).toHaveText('2'); + await page.getByRole('button', { name: /Payment gateway timeout/ }).click(); + const detail = page.getByRole('dialog'); + await expect(detail.getByTestId('error-occurrence-table').locator('tbody tr')).toHaveCount(2); + const links = detail.getByRole('link', { name: /Open trace/ }); + await expect(links).toHaveCount(2); + const href = new URL((await links.first().getAttribute('href'))!, 'http://localhost'); + expect(href.pathname).toBe('/traces/explore/orders_traces'); + expect(href.searchParams.get('start')).toBe(bounds.startTime); + expect(href.searchParams.get('end')).toBe(bounds.endTime); + expect(href.searchParams.get('traceId')).toBe('a'.repeat(32)); + await expect(detail.getByRole('link', { name: /SQL/ })).toHaveCount(0); +}); + +test('changing the range resets the issue page and uses one captured interval for all panels', async ({ + page, +}) => { + const fixture = await serveLogs(page); + await page.goto(`/errors/application_logs?${range}`); + await page.getByRole('button', { name: 'Next page', exact: true }).click(); + await expect.poll(() => fixture.requests.some((request) => request.offset === 25)).toBe(true); + const previousRequests = fixture.requests.length; + await page.getByLabel('Time range', { exact: true }).selectOption('6h'); + await expect(page).toHaveURL(/range=6h/); + await expect + .poll( + () => new Set(fixture.requests.slice(previousRequests).map((request) => request.kind)).size, + ) + .toBe(3); + const refreshed = fixture.requests.slice(previousRequests); + expect(refreshed.find((request) => request.kind === 'issues')?.offset).toBe(0); + expect(new Set(refreshed.map((request) => `${request.startTime}/${request.endTime}`)).size).toBe( + 1, + ); + expect(Date.parse(refreshed[0].endTime) - Date.parse(refreshed[0].startTime)).toBe(6 * 3_600_000); +}); + +test('sample log investigation runs in the SQL editor instead of handing off an unsupported query', async ({ + page, +}) => { + await page.addInitScript(() => sessionStorage.setItem('parseable-mode', 'demo')); + await page.goto('/errors/application_logs'); + await page + .getByRole('button', { name: /Upstream request timed out/ }) + .first() + .click(); + await page + .getByRole('dialog') + .getByRole('link', { name: /SQL editor/ }) + .click(); + await expect(page).toHaveURL(/\/sql-editor/); + await page.getByRole('button', { name: 'Run query', exact: true }).click(); + await expect(page.getByRole('table')).toBeVisible(); + await expect(page.getByRole('table')).toContainText('Upstream request timed out'); + await expect(page.getByText(/Demo SQL supports SELECT/)).toHaveCount(0); +}); diff --git a/frontend/e2e/operations.spec.ts b/frontend/e2e/operations.spec.ts index 7a7ab4f64..3caee7fa3 100644 --- a/frontend/e2e/operations.spec.ts +++ b/frontend/e2e/operations.spec.ts @@ -22,6 +22,18 @@ test('errors uses supported OTLP error span endpoint and opens trace detail', as const requests: unknown[] = []; await page.route('**/api/**', async (route) => { const path = new URL(route.request().url()).pathname; + if (path.endsWith('/schema')) + return route.fulfill({ + json: { + fields: [ + 'p_timestamp', + 'service.name', + 'span_trace_id', + 'span_span_id', + 'span_status_code', + ].map((name) => ({ name })), + }, + }); if (path === '/api/v1/logstream') return route.fulfill({ json: [{ name: 'orders_traces' }] }); if (path.endsWith('/info')) return route.fulfill({ @@ -40,7 +52,7 @@ test('errors uses supported OTLP error span endpoint and opens trace detail', as } return route.fulfill({ json: {} }); }); - await page.goto('/errors/orders_traces'); + await page.goto('/errors/orders_traces?view=spans'); await expect(page.getByRole('heading', { name: 'Errors', exact: true })).toBeVisible(); await expect(page.getByTestId('trace-table')).toBeVisible(); expect(requests[0]).toMatchObject({ @@ -86,6 +98,18 @@ test('errors explains ineligible datasets without querying fabricated error APIs let unsupportedRequests = 0; await page.route('**/api/**', async (route) => { const path = new URL(route.request().url()).pathname; + if (path.endsWith('/schema')) + return route.fulfill({ + json: { + fields: [ + 'p_timestamp', + 'service.name', + 'span_trace_id', + 'span_span_id', + 'span_status_code', + ].map((name) => ({ name })), + }, + }); if (path.includes('/errors/')) unsupportedRequests++; if (path === '/api/v1/logstream') return route.fulfill({ json: [{ name: 'logs' }] }); if (path.endsWith('/info')) @@ -108,6 +132,18 @@ test('errors retries unverified dataset metadata instead of declaring it ineligi let metadataUnavailable = true; await page.route('**/api/**', async (route) => { const path = new URL(route.request().url()).pathname; + if (path.endsWith('/schema')) + return route.fulfill({ + json: { + fields: [ + 'p_timestamp', + 'service.name', + 'span_trace_id', + 'span_span_id', + 'span_status_code', + ].map((name) => ({ name })), + }, + }); if (path === '/api/v1/logstream') return route.fulfill({ json: [{ name: 'orders_traces' }] }); if (path.endsWith('/info')) return metadataUnavailable @@ -119,7 +155,7 @@ test('errors retries unverified dataset metadata instead of declaring it ineligi return route.fulfill({ json: { count: 0, offset: 0, limit: 25, records: [] } }); return route.fulfill({ json: {} }); }); - await page.goto('/errors/orders_traces'); + await page.goto('/errors/orders_traces?view=spans'); await expect(page.getByText('Could not verify dataset', { exact: true })).toBeVisible(); await expect(page.getByText('No eligible error dataset')).toHaveCount(0); metadataUnavailable = false; diff --git a/frontend/src/features/errors/ErrorIssues.tsx b/frontend/src/features/errors/ErrorIssues.tsx new file mode 100644 index 000000000..ca85f7e88 --- /dev/null +++ b/frontend/src/features/errors/ErrorIssues.tsx @@ -0,0 +1,520 @@ +import { useCallback, useMemo, useState, type ReactNode } from 'react'; +import { Link } from 'react-router-dom'; +import { ArrowDown, ArrowUpDown, ArrowUpRight, ChevronLeft, ChevronRight } from 'lucide-react'; +import { useApp } from '../../app/AppProvider'; +import { useAsync } from '../../hooks/useAsync'; +import { Button, EmptyState, Sheet } from '../../components/ui'; +import { QueryState } from '../../components/explorer/QueryState'; +import { formatTimestamp } from '../../components/explorer/timezone'; +import { TimeSeriesChart } from '../../components/charts/TimeSeriesChart'; +import { + buildErrorExplorerSql, + ERROR_PAGE_SIZE, + loadErrorIssues, + loadErrorOccurrences, + loadErrorSeries, + loadErrorSummary, + type ErrorBounds, + type ErrorBucket, + type ErrorIssue, + type ErrorSchema, + type ErrorSort, +} from '../../lib/errorAnalytics'; +import { apmTransportBounds } from '../../lib/apm'; + +type Props = { dataset: string; schema: ErrorSchema; bounds: ErrorBounds }; + +/** Source-specific wording; a log record and an error span are different things. */ +export function occurrenceNoun(schema: ErrorSchema, count = 2): string { + const plural = count !== 1; + return schema.source === 'traces' + ? plural + ? 'error spans' + : 'error span' + : plural + ? 'error log records' + : 'error log record'; +} + +function utc(value: string | null): string { + return value ? `${formatTimestamp(value, 'UTC')} UTC` : '—'; +} + +function minutes(bounds: ErrorBounds): number { + return (Date.parse(bounds.endTime) - Date.parse(bounds.startTime)) / 60_000; +} + +function perMinute(count: number, bounds: ErrorBounds): string { + const rate = count / minutes(bounds); + return rate > 0 && rate < 0.01 + ? '<0.01' + : rate.toLocaleString(undefined, { maximumFractionDigits: 2 }); +} + +/** A missing value is shown as a placeholder that cannot be mistaken for a literal value. */ +function Value({ value, missing }: { value: string | null; missing: string }) { + return value === null ? {missing} : <>{value}; +} + +function fieldList(fields: string[]): string { + return fields.length ? fields.join(', ') : 'none in this dataset'; +} + +export function ErrorCountingNote({ schema }: { schema: ErrorSchema }) { + const severity = [ + schema.severity.text.length + ? `${schema.severity.text.join(' or ')} is an error level such as error, fatal or critical` + : '', + schema.severity.number.length + ? `${schema.severity.number.join(' or ')} is between 17 and 24 (OpenTelemetry error and fatal)` + : '', + schema.severity.text.includes('level') ? 'level is 50 or 60 (Pino/Bunyan error and fatal)' : '', + ].filter(Boolean); + return ( +
+ How errors are counted and grouped +

+ {schema.source === 'traces' + ? 'Each span with error status counts once, even if the same span was received more than once.' + : `Each log record counts once when ${severity.join(', or when ')}.`}{' '} + Times use{' '} + {schema.timeColumn === 'p_timestamp' ? 'ingestion time' : `the ${schema.timeColumn} field`}. +

+

+ Issues group occurrences that have exactly the same service, error type and message. + Messages that differ in any way, such as a request ID or number, are separate issues. +

+ {schema.exception.type.length + schema.exception.message.length > 0 && ( +

Exception details recorded on a span take priority over its status message.

+ )} +
+
Service
+
{fieldList(schema.service)}
+
Error type
+
{fieldList([...schema.exception.type, ...schema.errorType])}
+
Message
+
{fieldList([...schema.exception.message, ...schema.message])}
+
+
+ ); +} + +function FrequencyChart({ + title, + buckets, + bounds, + label, +}: { + title: string; + buckets: ErrorBucket[]; + bounds: ErrorBounds; + label: string; +}) { + return ( + Date.parse(bucket.bucket) / 1000)} + series={[{ id: 'occurrences', label, values: buckets.map((b) => b.occurrences) }]} + xRange={[Date.parse(bounds.startTime) / 1000, Date.parse(bounds.endTime) / 1000]} + kind="bar" + height={200} + timeZone="UTC" + showSeriesCount={false} + formatValue={(value) => value.toLocaleString()} + /> + ); +} + +function Pager({ + offset, + count, + total, + hasMore, + noun, + onOffset, +}: { + offset: number; + count: number; + total?: number; + hasMore: boolean; + noun: string; + onOffset: (offset: number) => void; +}) { + if (!count && !offset) return null; + const page = Math.floor(offset / ERROR_PAGE_SIZE) + 1; + return ( +
+ + Showing {offset + 1}–{offset + count} + {total !== undefined ? ` of ${total.toLocaleString()}` : ''} {noun} + +
+ + + Page {page} + {total !== undefined ? ` of ${Math.max(1, Math.ceil(total / ERROR_PAGE_SIZE))}` : ''} + + +
+
+ ); +} + +const sortColumns: { sort: ErrorSort; title: string }[] = [ + { sort: 'frequency', title: 'Occurrences' }, + { sort: 'latest', title: 'Last seen (UTC)' }, + { sort: 'first', title: 'First seen (UTC)' }, +]; + +function SortHeader({ + column, + sort, + onSort, +}: { + column: (typeof sortColumns)[number]; + sort: ErrorSort; + onSort: (sort: ErrorSort) => void; +}) { + const active = sort === column.sort; + const Icon = active ? ArrowDown : ArrowUpDown; + return ( + + + + ); +} + +function Stat({ label, value, note }: { label: string; value: ReactNode; note?: ReactNode }) { + return ( +
+
{label}
+
{value}
+ {note &&
{note}
} +
+ ); +} + +/** Grouped error issues for one dataset and one captured time range. The parent + * remounts this view when the dataset, range or refresh changes, resetting paging, + * sort order and the selected issue together. */ +export function ErrorIssues({ dataset, schema, bounds }: Props) { + const { client } = useApp(); + const [sort, setSort] = useState('frequency'); + const [offset, setOffset] = useState(0); + const [selected, setSelected] = useState(); + const options = { dataset, schema, bounds }; + const summary = useAsync( + useCallback( + (signal) => loadErrorSummary(client, { dataset, schema, bounds }, signal), + [client, dataset, schema, bounds], + ), + ); + const series = useAsync( + useCallback( + (signal) => loadErrorSeries(client, { dataset, schema, bounds }, signal), + [client, dataset, schema, bounds], + ), + ); + const issues = useAsync( + useCallback( + (signal) => loadErrorIssues(client, { dataset, schema, bounds, sort, offset }, signal), + [client, dataset, schema, bounds, sort, offset], + ), + ); + const noun = occurrenceNoun(schema); + const total = summary.data?.issues; + return ( +
+ + + {summary.data && ( +
+ + + + +
+ )} +
+ + {series.data && ( + + )} +
+
+

Issues

+ + {issues.data && + (issues.data.rows.length ? ( +
+
+ + + + + + + {sortColumns.map((column) => ( + { + setOffset(0); + setSort(value); + }} + /> + ))} + + + + {issues.data.rows.map((issue) => ( + + + + + + + + ))} + +
+ Error issues in {dataset}, grouped by exact service, error type and message +
IssueService
+ + + + {issue.occurrences.toLocaleString()}{utc(issue.lastSeen)}{utc(issue.firstSeen)}
+
+ +
+ ) : offset ? ( + setOffset(0)}>Back to first page} + /> + ) : ( + + ))} +
+ {selected && ( + setSelected(undefined)} /> + )} +
+ ); +} + +function ErrorIssueDetail({ + dataset, + schema, + bounds, + issue, + onClose, +}: Props & { issue: ErrorIssue; onClose: () => void }) { + const { client, mode } = useApp(); + const [offset, setOffset] = useState(0); + // Queries and handoffs identify the issue only by its exact group key. + const key = useMemo( + () => ({ service: issue.service, errorType: issue.errorType, message: issue.message }), + [issue], + ); + const series = useAsync( + useCallback( + (signal) => loadErrorSeries(client, { dataset, schema, bounds, issue: key }, signal), + [client, dataset, schema, bounds, key], + ), + ); + const occurrences = useAsync( + useCallback( + (signal) => + loadErrorOccurrences(client, { dataset, schema, bounds, issue: key, offset }, signal), + [client, dataset, schema, bounds, key, offset], + ), + ); + const range = { start: bounds.startTime, end: bounds.endTime }; + const traceLink = (traceId: string) => + `/traces/explore/${encodeURIComponent(dataset)}?${new URLSearchParams({ traceId, ...range })}`; + // The SQL keeps the exact range; the editor's request range is rounded down to + // whole minutes by the server, so hand off the enclosing minute envelope. + const envelope = apmTransportBounds(bounds); + const explorerSql = + schema.source === 'logs' ? buildErrorExplorerSql({ dataset, schema, bounds, issue: key }) : ''; + // The sample adapter needs its canonical query marker. Live users should see + // readable SQL without duplicated issue metadata in the editor or URL. + const editorSql = + mode === 'demo' ? explorerSql : explorerSql.replace(/^\/\* parseable-errors:[^\n]+ \*\/\n/, ''); + const sqlLink = + schema.source === 'logs' + ? `/sql-editor?${new URLSearchParams({ + query: editorSql, + start: envelope.startTime, + end: envelope.endTime, + })}` + : ''; + const traces = schema.source === 'traces'; + return ( + !open && onClose()} + title={issue.message ?? 'No message'} + description={`${issue.occurrences.toLocaleString()} ${occurrenceNoun(schema, issue.occurrences)} in the selected time range`} + className="errors-sheet" + > +
+
Service
+
+ +
+
Error type
+
+ +
+
Message
+
+ +
+
First seen
+
{utc(issue.firstSeen)}
+
Last seen
+
{utc(issue.lastSeen)}
+
+ {!traces && ( + + Open these log records in the SQL editor + ); +} diff --git a/frontend/src/features/errors/ErrorsPage.stories.tsx b/frontend/src/features/errors/ErrorsPage.stories.tsx index 92c94916a..4c41cc22a 100644 --- a/frontend/src/features/errors/ErrorsPage.stories.tsx +++ b/frontend/src/features/errors/ErrorsPage.stories.tsx @@ -4,79 +4,184 @@ import { createMemoryRouter, RouterProvider } from 'react-router-dom'; import { AppContext } from '../../app/AppProvider'; import { ApiError } from '../../lib/apiError'; import { createDemoClient } from '../../lib/demo'; -import type { ParseableClient, TraceListRequest } from '../../lib/types'; +import { handleErrorDemoQuery } from '../../lib/errorAnalyticsDemo'; +import type { LogRecord, ParseableClient, QueryRequest, TraceListRequest } from '../../lib/types'; import { storyTraceList, STORY_BOUNDS } from '../traces/fixtures'; import { ErrorsPage } from './ErrorsPage'; -type State = 'success' | 'loading' | 'error' | 'permission' | 'empty' | 'unavailable'; + +type State = + | 'success' + | 'logs' + | 'loading' + | 'error' + | 'permission' + | 'empty' + | 'unavailable' + | 'unsupported' + | 'unchecked'; + +const start = Date.parse(STORY_BOUNDS.startTime); +const at = (minute: number) => new Date(start + minute * 60_000 + 1_234).toISOString(); + +const TRACE_FIELDS = [ + 'p_timestamp', + 'service.name', + 'span_trace_id', + 'span_span_id', + 'span_name', + 'span_status_code', + 'span_status_message', + 'event_name', + 'event_exception.type', + 'event_exception.message', +]; +/** Raw span rows: a retried span and its exception event count once. */ +const traceRows: LogRecord[] = [ + ...Array.from({ length: 9 }, (_, index) => ({ + p_timestamp: at(index * 6), + 'service.name': 'checkout', + span_trace_id: `${index.toString(16)}`.padStart(32, 'a'), + span_span_id: `${index.toString(16)}`.padStart(16, '1'), + span_name: 'authorize-payment', + span_status_code: 2, + span_status_message: 'Payment provider timed out', + event_name: index === 0 ? 'exception' : null, + 'event_exception.type': index === 0 ? 'TimeoutError' : null, + 'event_exception.message': index === 0 ? 'Card authorization deadline exceeded' : null, + })), + { + p_timestamp: at(20), + 'service.name': 'inventory', + span_trace_id: 'b'.repeat(32), + span_span_id: '2'.repeat(16), + span_name: 'reserve-stock', + span_status_code: 2, + span_status_message: '', + }, + { + p_timestamp: at(30), + 'service.name': 'inventory', + span_trace_id: 'c'.repeat(32), + span_span_id: '3'.repeat(16), + span_name: 'reserve-stock', + span_status_code: 1, + span_status_message: '', + }, +]; + +const LOG_FIELDS = [ + 'p_timestamp', + 'level', + 'severity_number', + 'service', + 'message', + 'exception.type', + 'trace_id', +]; +const logRows: LogRecord[] = [ + ...Array.from({ length: 31 }, (_, index) => ({ + p_timestamp: at(index % 58), + level: index % 4 ? 'ERROR' : 'fatal', + service: 'api-gateway', + message: 'Upstream request timed out after 3000ms', + 'exception.type': 'UpstreamTimeout', + trace_id: `8e74f620af734b9086bb${index.toString(16).padStart(12, '0')}`, + })), + // Exact grouping: a different request ID is a different issue. + ...Array.from({ length: 27 }, (_, index) => ({ + p_timestamp: at(40 + (index % 15)), + severity_number: 17, + service: 'worker', + message: `Job ${1000 + index} failed`, + })), + { p_timestamp: at(12), level: 'ERROR', message: 'Unhandled rejection' }, + { p_timestamp: at(13), level: 'INFO', service: 'worker', message: 'Job started' }, +]; + function Example({ state }: { state: State }) { - const [requests, setRequests] = useState([]); - const client = useMemo( - () => - ({ - ...createDemoClient(), - async listDatasets() { - return state === 'unavailable' - ? [] - : [{ name: 'orders_traces', type: 'traces' as const }]; - }, - async datasetInfo(name: string) { - return { name, telemetryType: 'traces', logSourceFormats: ['otel-traces'] }; - }, - async listTraces(request: TraceListRequest, signal?: AbortSignal) { - setRequests((previous) => [...previous, request]); - signal?.throwIfAborted(); - if (state === 'loading') - await new Promise((_, reject) => - signal?.addEventListener( - 'abort', - () => reject(new DOMException('Aborted', 'AbortError')), - { once: true }, - ), - ); - if (state === 'error') throw new ApiError('Trace query node unavailable', 503); - if (state === 'permission') throw new ApiError('Error span access denied', 403); - const rows = state === 'empty' ? [] : storyTraceList.filter((row) => row.error_count > 0); - const sorts = { - mostRecent: ['span_start_time_unix_nano', -1], - leastRecent: ['span_start_time_unix_nano', 1], - longestFirst: ['span_duration_ns', -1], - shortestFirst: ['span_duration_ns', 1], - mostSpans: ['total_span_count', -1], - leastSpans: ['total_span_count', 1], - } as const; - const [field, direction] = sorts[request.sortBy ?? 'mostRecent']; - rows.sort( - (a, b) => - (field === 'span_start_time_unix_nano' - ? Date.parse(a[field]) - Date.parse(b[field]) - : Number(a[field]) - Number(b[field])) * direction, - ); - return { - count: rows.length, - offset: request.offset ?? 0, - limit: request.limit ?? 25, - records: rows.slice(request.offset ?? 0, (request.offset ?? 0) + (request.limit ?? 25)), - }; - }, - }) satisfies ParseableClient, - [state], - ); + const [traceRequests, setTraceRequests] = useState([]); + const [queries, setQueries] = useState([]); + const client = useMemo(() => { + const pending = (signal?: AbortSignal) => + new Promise((_, reject) => + signal?.addEventListener('abort', () => reject(new DOMException('Aborted', 'AbortError')), { + once: true, + }), + ); + const fail = () => { + if (state === 'error') throw new ApiError('Query node unavailable', 503); + if (state === 'permission') throw new ApiError('Error data access denied', 403); + }; + return { + ...createDemoClient(), + async listDatasets() { + return state === 'unavailable' + ? [] + : [ + { name: 'orders_logs', type: 'logs' as const }, + { name: 'orders_traces', type: 'traces' as const }, + ]; + }, + async datasetInfo(name: string) { + if (state === 'unchecked' && name === 'orders_traces') + throw new ApiError('Metadata service unavailable', 503); + return name === 'orders_traces' && state !== 'unsupported' + ? { name, telemetryType: 'traces', logSourceFormats: ['otel-traces'] } + : { name, telemetryType: 'logs', logSourceFormats: ['json'] }; + }, + async schema(name: string) { + if (state === 'unsupported') return ['p_timestamp', 'message']; + return name === 'orders_traces' ? TRACE_FIELDS : LOG_FIELDS; + }, + async query(request: QueryRequest, signal?: AbortSignal) { + setQueries((previous) => [...previous, request]); + signal?.throwIfAborted(); + if (state === 'loading') return pending(signal); + fail(); + const rows = + state === 'empty' ? [] : request.sql.includes('"orders_traces"') ? traceRows : logRows; + const result = handleErrorDemoQuery(request, rows); + if (!result) throw new ApiError('Unsupported story query', 400); + return result; + }, + async listTraces(request: TraceListRequest, signal?: AbortSignal) { + setTraceRequests((previous) => [...previous, request]); + signal?.throwIfAborted(); + if (state === 'loading') return pending(signal); + fail(); + const rows = state === 'empty' ? [] : storyTraceList.filter((row) => row.error_count > 0); + const sorts = { + mostRecent: ['span_start_time_unix_nano', -1], + leastRecent: ['span_start_time_unix_nano', 1], + longestFirst: ['span_duration_ns', -1], + shortestFirst: ['span_duration_ns', 1], + mostSpans: ['total_span_count', -1], + leastSpans: ['total_span_count', 1], + } as const; + const [field, direction] = sorts[request.sortBy ?? 'mostRecent']; + rows.sort( + (a, b) => + (field === 'span_start_time_unix_nano' + ? Date.parse(a[field]) - Date.parse(b[field]) + : Number(a[field]) - Number(b[field])) * direction, + ); + return { + count: rows.length, + offset: request.offset ?? 0, + limit: request.limit ?? 25, + records: rows.slice(request.offset ?? 0, (request.offset ?? 0) + (request.limit ?? 25)), + }; + }, + } satisfies ParseableClient; + }, [state]); const router = useMemo( () => - createMemoryRouter( - [ - { - path: '/errors/:dataset?', - element: , - }, + createMemoryRouter([{ path: '/errors/:dataset?', element: }], { + initialEntries: [ + `/errors/${state === 'logs' ? 'orders_logs' : 'orders_traces'}?${new URLSearchParams({ start: STORY_BOUNDS.startTime, end: STORY_BOUNDS.endTime })}`, ], - { - initialEntries: [ - `/errors/orders_traces?${new URLSearchParams({ start: STORY_BOUNDS.startTime, end: STORY_BOUNDS.endTime })}`, - ], - }, - ), - [], + }), + [state], ); return (
Story transport inspection - 
{JSON.stringify(requests, null, 2)}
+ 
{JSON.stringify(traceRequests, null, 2)}
+ 
{JSON.stringify(queries, null, 2)}
 ); } + const meta = { title: 'Operations/Errors', component: Example } satisfies Meta; export default meta; type Story = StoryObj; export const Success: Story = { args: { state: 'success' } }; +export const Logs: Story = { args: { state: 'logs' } }; +export const TraceIssueDetail: Story = { + args: { state: 'success' }, + play: async ({ canvas, userEvent }) => { + // Issues are ordered by frequency; open the most frequent one. + await userEvent.click((await canvas.findAllByRole('button', { name: /Payment provider/ }))[0]); + }, +}; +export const LogIssueDetail: Story = { + args: { state: 'logs' }, + play: async ({ canvas, userEvent }) => { + await userEvent.click((await canvas.findAllByRole('button', { name: /Upstream request/ }))[0]); + }, +}; export const Loading: Story = { args: { state: 'loading' } }; export const Error: Story = { args: { state: 'error' } }; export const Permission: Story = { args: { state: 'permission' } }; export const Empty: Story = { args: { state: 'empty' } }; export const Unavailable: Story = { args: { state: 'unavailable' } }; +export const UnsupportedSchema: Story = { args: { state: 'unsupported' } }; +export const MetadataUnavailable: Story = { args: { state: 'unchecked' } }; diff --git a/frontend/src/features/errors/ErrorsPage.tsx b/frontend/src/features/errors/ErrorsPage.tsx index 7a9129ab2..2b37ac3af 100644 --- a/frontend/src/features/errors/ErrorsPage.tsx +++ b/frontend/src/features/errors/ErrorsPage.tsx @@ -1,24 +1,39 @@ -import { useCallback, useMemo, useState } from 'react'; -import { Link, useNavigate, useParams, useSearchParams } from 'react-router-dom'; +import { useCallback, useEffect, useMemo, useState } from 'react'; +import { Link, useLocation, useNavigate, useSearchParams } from 'react-router-dom'; +import { RefreshCw } from 'lucide-react'; import { useApp } from '../../app/AppProvider'; import { useAsync } from '../../hooks/useAsync'; import { PageHeader } from '../../components/explorer/PageHeader'; import { QueryState } from '../../components/explorer/QueryState'; -import { Badge, Button, EmptyState, Select } from '../../components/ui'; -import { discoverTraceDatasets, TRACE_PAGE_SIZE } from '../traces/helpers'; +import { Badge, Button, EmptyState, Select, Tabs } from '../../components/ui'; +import { TRACE_PAGE_SIZE } from '../traces/helpers'; import { TraceTable } from '../traces/TraceTable'; import { TimeRangePicker } from '../../components/explorer/TimeRangePicker'; -import { paramsRange, timeBounds } from '../../lib/query'; +import { apmBoundsFromParams, apmRangeFromParams, resolveApmRoute } from '../../lib/apm'; +import { ApiError } from '../../lib/apiError'; +import { + discoverErrorDatasets, + loadErrorSchema, + UnsupportedErrorSchema, + type ErrorBounds, + type ErrorSchema, +} from '../../lib/errorAnalytics'; +import { timeBounds } from '../../lib/query'; import type { TimeRange, TraceSortBy } from '../../lib/types'; +import { ErrorIssues } from './ErrorIssues'; import '../traces/traces.css'; -export function ErrorsUnavailable({ dataset }: { dataset?: string }) { +import './errors.css'; + +const APPLIED_PARAMS = ['start', 'end', 'range', 'view']; + +export function ErrorsUnavailable({ dataset, reason }: { dataset?: string; reason?: string }) { return ( ); } -function errorRange(params: URLSearchParams): TimeRange { - if (params.has('start') || params.has('end')) return paramsRange(params); - const preset = params.get('range'); - if (preset) { - try { - timeBounds(preset as TimeRange); - return preset as TimeRange; - } catch { - /* Fall back to the default range. */ - } - } - return '1h'; -} -function ErrorSpans({ dataset }: { dataset: string }) { + +function ErrorSpans({ dataset, bounds }: { dataset: string; bounds: ErrorBounds }) { const { client } = useApp(); const navigate = useNavigate(); - const [params, setParams] = useSearchParams(); - const rangeKey = JSON.stringify(errorRange(params)); - const range = useMemo(() => JSON.parse(rangeKey) as TimeRange, [rangeKey]); const [page, setPage] = useState(0); const [sort, setSort] = useState('mostRecent'); - const [revision, setRevision] = useState(0); - const bounds = useMemo(() => timeBounds(range), [range, revision]); - function changeRange(value: TimeRange) { - const next = new URLSearchParams(params); - next.delete('start'); - next.delete('end'); - next.delete('range'); - if (typeof value === 'string') next.set('range', value); - else { - next.set('start', value.startTime); - next.set('end', value.endTime); - } - setPage(0); - setSort('mostRecent'); - setParams(next); - } const result = useAsync( useCallback( (signal) => @@ -87,18 +71,9 @@ function ErrorSpans({ dataset }: { dataset: string }) { ); return ( <> -
-

OTLP error spans · {TRACE_PAGE_SIZE} results per page

- - -
+

+ Individual error spans, {TRACE_PAGE_SIZE} per page. Open one to see its whole trace. +

{result.data && (result.data.records.length ? ( @@ -126,76 +101,188 @@ function ErrorSpans({ dataset }: { dataset: string }) { ); } + +/** Explains why a dataset outside the discovered list cannot be analyzed, without + * hiding permission or transport failures behind an "unsupported" message. */ +function DatasetUnavailable({ dataset }: { dataset: string }) { + const { client } = useApp(); + const check = useAsync( + useCallback((signal) => loadErrorSchema(client, dataset, signal), [client, dataset]), + ); + if (check.loading) return ; + const unsupported = check.error instanceof UnsupportedErrorSchema; + const missing = check.error instanceof ApiError && check.error.status === 404; + if (check.error && !unsupported && !missing) + return ; + return ( + + ); +} + +function ErrorDataset({ + dataset, + schema, + bounds, + view, + onView, +}: { + dataset: string; + schema: ErrorSchema; + bounds: ErrorBounds; + view: string; + onView: (view: string) => void; +}) { + const issues = ; + if (schema.source !== 'traces') return issues; + return ( + , + }, + ]} + /> + ); +} + export function ErrorsPage() { - const { dataset } = useParams(); - const [params] = useSearchParams(); const { client, mode } = useApp(); + const location = useLocation(); + // Router params decode %2F before matching; decode the raw segment exactly once. + const { dataset } = resolveApmRoute(location.pathname); + const [params, setParams] = useSearchParams(); const navigate = useNavigate(); + const [refreshRevision, setRefreshRevision] = useState(0); + const searchString = params.toString(); + const bounds = useMemo( + () => apmBoundsFromParams(new URLSearchParams(searchString)), + [searchString], + ); + const range = useMemo( + () => apmRangeFromParams(new URLSearchParams(searchString)), + [searchString], + ); + // Capture a relative range once so paging, sorting, details and handoffs share it. + useEffect(() => { + if (params.get('start') && params.get('end')) return; + const next = new URLSearchParams(params); + next.set('start', bounds.startTime); + next.set('end', bounds.endTime); + if (typeof range === 'string') next.set('range', range); + setParams(next, { replace: true }); + }, [params, bounds, range, setParams]); const discovery = useAsync( - useCallback((signal) => discoverTraceDatasets(client, signal), [client]), + useCallback((signal) => discoverErrorDatasets(client, signal), [client]), + { keepPreviousData: true }, ); - const selected = dataset ?? discovery.data?.datasets[0]?.name; - const eligible = discovery.data?.datasets.some((entry) => entry.name === selected); + const changeRange = (value: TimeRange) => { + const captured = timeBounds(value), + next = new URLSearchParams(params); + next.set('start', captured.startTime); + next.set('end', captured.endTime); + if (typeof value === 'string') next.set('range', value); + else next.delete('range'); + setParams(next); + }; + const refresh = () => { + changeRange(range); + setRefreshRevision((revision) => revision + 1); + }; + const changeView = (view: string) => { + const next = new URLSearchParams(params); + if (view === 'issues') next.delete('view'); + else next.set('view', view); + setParams(next, { replace: true }); + }; + const datasets = discovery.data?.datasets ?? []; + const selected = dataset ?? datasets[0]?.name; + const entry = datasets.find((candidate) => candidate.name === selected); + const unapplied = [ + ...new Set( + [...params.keys()].filter( + (key) => + !APPLIED_PARAMS.includes(key) || + (key === 'view' && + (entry?.source === 'logs' + ? params.get('view') !== 'issues' + : !['issues', 'spans'].includes(params.get('view') ?? ''))), + ), + ), + ]; return ( -
- - {mode === 'demo' && Sample data} -

- This view lists individual error spans. Grouped error issues and frequency analysis are - unavailable on this server. -

- {[...params.keys()].some((key) => !['start', 'end', 'range'].includes(key)) && ( -

- This link includes options that are not applied here:{' '} - {[...new Set(params.keys())] - .filter((key) => !['start', 'end', 'range'].includes(key)) - .join(', ')} - . Use the native time range and sort controls to refine these error spans. +

+ Sample data : undefined} + /> + {unapplied.length > 0 && ( +

+ This link includes options that are not applied here: {unapplied.join(', ')}.

)} +
+ + + +
{discovery.data?.unchecked.length ? ( -

+

Could not check datasets: {discovery.data.unchecked.join(', ')}.{' '} - +

) : null} - {discovery.data && ( - <> - - {eligible && selected ? ( - - ) : selected && discovery.data.unchecked.includes(selected) ? ( - Retry dataset verification} - /> - ) : ( - - )} - - )} + {discovery.data && + (entry ? ( + + ) : selected && discovery.data.unchecked.includes(selected) ? ( + Retry dataset verification} + /> + ) : dataset ? ( + + ) : ( + + ))}
); } diff --git a/frontend/src/features/errors/errors.css b/frontend/src/features/errors/errors.css new file mode 100644 index 000000000..cc423bafb --- /dev/null +++ b/frontend/src/features/errors/errors.css @@ -0,0 +1,166 @@ +.errors-page { + display: flex; + flex-direction: column; + gap: 20px; + min-width: 0; +} +.errors-page .notice { + margin: 0; +} +.errors-controls { + display: flex; + align-items: flex-end; + flex-wrap: wrap; + gap: 12px; + padding: 14px 18px; + border: 1px solid var(--color-border); + border-radius: 10px; + background: var(--color-surface); +} +.errors-controls .ui-field:first-child { + min-width: 220px; + margin-right: auto; +} +.errors-issues { + display: flex; + flex-direction: column; + gap: 16px; + min-width: 0; +} +.errors-caption, +.errors-note { + color: var(--color-muted); + font-size: 12px; + margin: 0; +} +.errors-note summary { + cursor: pointer; + color: var(--color-text); + font-size: 13px; +} +.errors-note p { + margin: 8px 0 0; + max-width: 80ch; +} +.errors-fields, +.errors-detail-fields { + display: grid; + grid-template-columns: max-content minmax(0, 1fr); + gap: 6px 16px; + margin: 8px 0 0; +} +.errors-fields dt, +.errors-detail-fields dt { + color: var(--color-muted); +} +.errors-fields dd, +.errors-detail-fields dd { + margin: 0; + overflow-wrap: anywhere; +} +.errors-detail-fields { + margin: 0 0 16px; + font-size: 13px; +} +.errors-detail-message { + white-space: pre-wrap; +} +.errors-stat-grid { + display: grid; + gap: 12px; + grid-template-columns: repeat(4, minmax(0, 1fr)); + margin: 0; +} +.errors-stat { + padding: 18px 20px; + background: var(--color-surface); + border: 1px solid var(--color-border); + border-radius: 10px; + min-width: 0; +} +.errors-stat dt { + color: var(--color-muted); + font-size: 12px; + margin-bottom: 10px; +} +.errors-stat dd { + margin: 0; + font-size: 22px; + font-weight: 600; + overflow-wrap: anywhere; +} +.errors-stat dd.errors-stat-note { + color: var(--color-muted); + margin-top: 8px; + font-size: 11px; + font-weight: 400; +} +.errors-panel { + min-width: 0; + padding: 16px 18px; + border: 1px solid var(--color-border); + border-radius: 10px; + background: var(--color-surface); +} +.errors-panel h2 { + font-size: 15px; + margin: 0 0 12px; +} +.errors-sheet section { + margin-top: 20px; +} +.errors-sheet h3 { + font-size: 14px; + margin: 0 0 10px; +} +.errors-issue-button { + display: flex; + flex-direction: column; + align-items: flex-start; + gap: 3px; + max-width: 60ch; + padding: 0; + border: 0; + background: none; + color: inherit; + font: inherit; + text-align: left; + cursor: pointer; +} +.errors-issue-button:hover .errors-issue-message { + text-decoration: underline; +} +.errors-issue-button:focus-visible { + outline: 2px solid var(--color-focus, currentColor); + outline-offset: 2px; +} +.errors-issue-message { + font-weight: 500; + overflow-wrap: anywhere; +} +.errors-issue-type { + color: var(--color-muted); + font-size: 12px; +} +.errors-missing { + color: var(--color-muted); + font-style: italic; +} +.errors-number { + font-variant-numeric: tabular-nums; + text-align: right; +} +@media (max-width: 1000px) { + .errors-stat-grid { + grid-template-columns: repeat(2, minmax(0, 1fr)); + } +} +@media (max-width: 600px) { + .errors-stat-grid { + grid-template-columns: minmax(0, 1fr); + } + .errors-controls .ui-field:first-child { + min-width: 0; + width: 100%; + } +} diff --git a/frontend/src/features/sql/SqlPage.tsx b/frontend/src/features/sql/SqlPage.tsx index 423249b77..0fd4400c8 100644 --- a/frontend/src/features/sql/SqlPage.tsx +++ b/frontend/src/features/sql/SqlPage.tsx @@ -185,6 +185,8 @@ function SqlWorkspace() { return; } const alreadyProcessed = url === previousUrl.current; + // Keep the handoff marked as processed until the router commits its removal. + // An edit can rerun this effect while params still contains the old query. previousUrl.current = url; const query = params.get('query') || ''; const range = paramsRange(params); @@ -200,7 +202,6 @@ function SqlWorkspace() { setRow(undefined); const consumed = new URLSearchParams(params); for (const key of ['query', 'start', 'end', 'range']) consumed.delete(key); - previousUrl.current = consumed.toString(); setParams(consumed, { replace: true }); }, [params, tabs, setParams]); useEffect(() => { diff --git a/frontend/src/lib/demo.ts b/frontend/src/lib/demo.ts index 6dd014e9b..674aee79b 100644 --- a/frontend/src/lib/demo.ts +++ b/frontend/src/lib/demo.ts @@ -19,6 +19,7 @@ import { createDemoDatasets } from './demoDatasets'; import { createDemoLogMethods } from './demoLogs'; import { createDemoTraceRecords, createDemoTraceMethods } from './demoTraces'; import { handleApmDemoQuery } from './apmDemo'; +import { errorDemoQueryTarget, handleErrorDemoQuery } from './errorAnalyticsDemo'; import { alertDuration } from '../features/alerts/helpers'; import { validateName } from '../features/team/helpers'; @@ -711,6 +712,22 @@ export function createDemoClient(): ParseableClient { const result = handleApmDemoQuery(request, item.name === 'demo_traces' ? traceRows : []); if (result !== undefined) return result; } + const errors = errorDemoQueryTarget(request); + if (errors) { + if (!current.some((dataset) => dataset.name === errors.dataset)) + throw new ApiError(`Unknown demo dataset: ${errors.dataset}`, 404); + const fields = new Set(await datasets.schema(errors.dataset, signal)); + const missing = errors.fields.filter((field) => !fields.has(field)); + if (missing.length) throw new ApiError(`Unknown demo field: ${missing[0]}`, 400); + // Seeded datasets share the fixture records; created datasets start empty. + const rows = + errors.dataset === 'demo_traces' + ? traceRows + : demoDatasets.some((dataset) => dataset.name === errors.dataset) + ? records + : []; + return handleErrorDemoQuery(request, rows)!; + } return executeDemoQuery( request, /FROM\s+(?:"demo_traces"|demo_traces\b)/i.test(request.sql) ? traceRows : records, diff --git a/frontend/src/lib/errorAnalytics.live.test.ts b/frontend/src/lib/errorAnalytics.live.test.ts new file mode 100644 index 000000000..7362bd6d2 --- /dev/null +++ b/frontend/src/lib/errorAnalytics.live.test.ts @@ -0,0 +1,459 @@ +import { randomUUID } from 'node:crypto'; +import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest'; +import { + buildErrorExplorerSql, + loadErrorSchema, + loadErrorSummary, + loadErrorSeries, + loadErrorIssues, + loadErrorOccurrences, + type ErrorSchema, +} from './errorAnalytics'; +import { apmTransportBounds } from './apm'; +import type { DatasetInfo } from './types'; + +// Explicitly opt in against a disposable Parseable server. Every dataset belongs +// to this run and is removed even when fixture setup or an assertion fails. +// ERRORS_REFERENCE_URL=http://127.0.0.1:8450 npx vitest run src/lib/errorAnalytics.live.test.ts +// Override ERRORS_REFERENCE_USERNAME/PASSWORD for another disposable server. +const origin = process.env.ERRORS_REFERENCE_URL?.replace(/\/$/, ''); +const username = process.env.ERRORS_REFERENCE_USERNAME ?? 'fixture'; +const password = process.env.ERRORS_REFERENCE_PASSWORD ?? 'fixture-throwaway'; +const headers = { + authorization: `Basic ${Buffer.from(`${username}:${password}`).toString('base64')}`, + 'content-type': 'application/json', +}; +const suffix = `${Date.now().toString(36)}_${randomUUID().slice(0, 8)}`; +const logDataset = `errors_logs_${suffix}`; +const traceDataset = `errors_traces_${suffix}`; +const numericDataset = `errors_otlp_logs_${suffix}`; +const levelDataset = `errors_pino_logs_${suffix}`; +const pairDataset = `errors_trace_pairs_${suffix}`; +const datasets = [logDataset, traceDataset, numericDataset, levelDataset, pairDataset]; +const attemptedDatasets = new Set(); +const escapedService = "checkout's / payments"; +const escapedMessage = "can't charge customer '42'"; +const escapedType = "Card'Declined"; +let bounds: { startTime: string; endTime: string }; +const schemas = new Map(); + +async function api(path: string, method: string, body?: unknown): Promise { + const response = await fetch(`${origin}${path}`, { + method, + headers, + body: body === undefined ? undefined : JSON.stringify(body), + signal: AbortSignal.timeout(10_000), + }); + if (!response.ok) + throw new Error(`${method} ${path}: ${response.status} ${await response.text()}`); + const text = await response.text(); + return text ? JSON.parse(text) : undefined; +} + +async function query(request: { sql: string; startTime: string; endTime: string }) { + return (await api('/api/v1/query', 'POST', { + query: request.sql, + startTime: request.startTime, + endTime: request.endTime, + sendNull: true, + })) as Record[]; +} + +const client = { + query, + async schema(dataset: string) { + const result = (await api(`/api/v1/logstream/${dataset}/schema`, 'GET')) as { + fields: { name: string }[]; + }; + return result.fields.map((field) => field.name); + }, + async datasetInfo(dataset: string): Promise { + const result = (await api(`/api/v1/logstream/${dataset}/info`, 'GET')) as { + logSource?: { log_source_format: string }[]; + timePartition?: string; + }; + return { + name: dataset, + logSourceFormats: result.logSource?.map((source) => source.log_source_format) ?? [], + timePartition: result.timePartition, + }; + }, +}; + +const options = (dataset: string) => ({ dataset, schema: schemas.get(dataset)!, bounds }); +const escapedIssue = { service: escapedService, errorType: escapedType, message: escapedMessage }; + +async function ingest(dataset: string, path: string, body: unknown) { + attemptedDatasets.add(dataset); + const response = await fetch(`${origin}${path}`, { + method: 'POST', + headers: { ...headers, 'X-P-Stream': dataset }, + body: JSON.stringify(body), + signal: AbortSignal.timeout(10_000), + }); + expect(response.ok, await response.text()).toBe(true); +} + +const attribute = (key: string, value: string) => ({ key, value: { stringValue: value } }); +const traceId = 'a'.repeat(32); +const spanId = (id: number) => id.toString(16).padStart(16, '0'); + +describe.skipIf(!origin)('Errors analytics on a real Parseable server', () => { + beforeAll(async () => { + const now = Date.now(); + const common = { + 'service.name': escapedService, + 'exception.type': escapedType, + message: escapedMessage, + }; + // Three identical error logs remain three occurrences; logs have no span + // identity with which to deduplicate them. Different services stay separate. + const logs: Record[] = [ + { ...common, level: 'ERROR' }, + { ...common, severity: 'fatal' }, + { ...common, severity_number: 17 }, + { ...common, 'service.name': 'other-service', level: 'error' }, + { level: 'error', message: 'missing service and type' }, + { level: 'info', message: 'ordinary information' }, + { level: 'WARN', message: 'warning is not an error' }, + // More than one page of distinct issues catches accidental LIMIT-before- + // aggregate bugs and makes the summary independent of the issue page. + ...Array.from({ length: 27 }, (_, index) => ({ + level: 'ERROR', + 'service.name': 'pagination', + message: `distinct failure ${String(index).padStart(2, '0')}`, + })), + ]; + await ingest(logDataset, '/api/v1/ingest', logs); + // pino/bunyan numeric levels: error (50) and fatal (60) count; warn (40), + // info (30) and the fractional 50.9 do not. + await ingest( + levelDataset, + '/api/v1/ingest', + [50, 60, 40, 30, 50.9].map((level) => ({ level, msg: `pino level ${level}` })), + ); + + const nanos = BigInt(now) * 1_000_000n; + // A separate actual OTLP log stream checks all eight error/fatal values and + // the server's flattened body/severity fields. The JSON fixture above also + // includes a numeric-only row, exercising the query's numeric predicate. + await ingest(numericDataset, '/v1/logs', { + resourceLogs: [ + { + resource: { attributes: [attribute('service.name', 'severity-numbers')] }, + scopeLogs: [ + { + scope: { name: 'errors-live-regression' }, + logRecords: [ + ...Array.from({ length: 9 }, (_, index) => ({ + timeUnixNano: String(nanos), + severityNumber: 16 + index, + severityText: index ? 'ERROR' : 'WARN', + body: { stringValue: `OTLP severity ${16 + index}` }, + })), + // The server derives ERROR/FATAL categories from these bodies, + // but unspecified severity must not qualify them as errors. + ...['0 errors found', 'no fatal failures'].map((body) => ({ + timeUnixNano: String(nanos), + severityNumber: 0, + body: { stringValue: body }, + })), + ], + }, + ], + }, + ], + }); + const span = (id: number, error: boolean, events = false) => ({ + traceId, + spanId: spanId(id), + name: `operation-${id}`, + kind: 2, + startTimeUnixNano: String(nanos - 10_000_000n), + endTimeUnixNano: String(nanos), + status: { code: error ? 2 : 1, message: error && !events ? escapedMessage : '' }, + attributes: error && !events ? [attribute('exception.type', escapedType)] : [], + events: events + ? [ + { + timeUnixNano: String(nanos), + name: 'exception', + attributes: [ + attribute('exception.type', escapedType), + attribute('exception.message', escapedMessage), + ], + }, + { + timeUnixNano: String(nanos), + name: 'exception', + attributes: [ + attribute('exception.type', escapedType), + attribute('exception.message', escapedMessage), + ], + }, + ] + : [], + }); + const envelope = (spans: ReturnType[]) => ({ + resourceSpans: [ + { + resource: { attributes: [attribute('service.name', escapedService)] }, + scopeSpans: [{ scope: { name: 'errors-live-regression' }, spans }], + }, + ], + }); + // The first error carries its type/message only on exception events, expands + // into event rows and is retried in a second request. The second uses span + // attributes/status. Both must group together as two distinct error spans. + await ingest( + traceDataset, + '/v1/traces', + envelope([span(1, true, true), span(2, true), span(3, false)]), + ); + await ingest(traceDataset, '/v1/traces', envelope([span(1, true, true)])); + // One erroneous span with two different exception events (plus direct span + // metadata), sent twice. Its type and message must come from the same event. + const exceptionEvent = (type: string, message: string) => ({ + timeUnixNano: String(nanos), + name: 'exception', + attributes: [attribute('exception.type', type), attribute('exception.message', message)], + }); + const twoExceptions = { + ...span(9, true), + status: { code: 2, message: 'status message' }, + attributes: [attribute('exception.type', 'DirectType')], + events: [exceptionEvent('A', 'zeta'), exceptionEvent('B', 'alpha')], + }; + for (let attempt = 0; attempt < 2; attempt++) + await ingest(pairDataset, '/v1/traces', envelope([twoExceptions])); + + const wideBounds = { + startTime: new Date(now - 60_000).toISOString(), + endTime: new Date(now + 120_000).toISOString(), + }; + for (const dataset of datasets) { + await vi.waitFor( + async () => { + const rows = await query({ + ...wideBounds, + sql: `SELECT p_timestamp FROM "${dataset}" LIMIT 1`, + }); + expect(rows, `${dataset} should expose its persisted fixture`).toHaveLength(1); + }, + { timeout: 30_000, interval: 500 }, + ); + } + const timestamps = await Promise.all( + datasets.map(async (dataset) => { + const rows = await query({ + ...wideBounds, + sql: `SELECT MIN(p_timestamp) AS first, MAX(p_timestamp) AS last FROM "${dataset}"`, + }); + const utc = (value: unknown) => + Date.parse(`${value}${String(value).endsWith('Z') ? '' : 'Z'}`); + return [utc(rows[0].first), utc(rows[0].last)]; + }), + ); + bounds = { + startTime: new Date(Math.min(...timestamps.map(([first]) => first))).toISOString(), + endTime: new Date(Math.max(...timestamps.map(([, last]) => last)) + 1).toISOString(), + }; + for (const dataset of datasets) schemas.set(dataset, await loadErrorSchema(client, dataset)); + }, 60_000); + + it('groups log errors by their exact service, type and message while excluding information and warnings', async () => { + const summary = await loadErrorSummary(client, options(logDataset)); + expect(summary).toMatchObject({ occurrences: 32, issues: 30 }); + const page = await loadErrorIssues(client, options(logDataset)); + expect(page.rows[0]).toMatchObject({ ...escapedIssue, occurrences: 3 }); + const series = await loadErrorSeries(client, options(logDataset)); + expect(series.reduce((sum, bucket) => sum + bucket.occurrences, 0)).toBe(32); + }); + + it('paginates all thirty issues without changing the full-range summary or repeating a group', async () => { + const first = await loadErrorIssues(client, options(logDataset)); + const second = await loadErrorIssues(client, { ...options(logDataset), offset: 25 }); + expect(first.rows).toHaveLength(25); + expect(first.hasMore).toBe(true); + expect(second.rows).toHaveLength(5); + expect(second.hasMore).toBe(false); + const all = [...first.rows, ...second.rows]; + expect(new Set(all.map((issue) => issue.id)).size).toBe(30); + expect(all.reduce((sum, issue) => sum + issue.occurrences, 0)).toBe(32); + expect(all).toContainEqual( + expect.objectContaining({ + service: null, + errorType: null, + message: 'missing service and type', + occurrences: 1, + }), + ); + expect(all).toContainEqual( + expect.objectContaining({ + ...escapedIssue, + service: 'other-service', + occurrences: 1, + }), + ); + }); + + it('executes escaped issue filters for both matching occurrences and matching frequency', async () => { + const page = await loadErrorOccurrences(client, { + ...options(logDataset), + issue: escapedIssue, + }); + expect(page.rows).toHaveLength(3); + expect(page.hasMore).toBe(false); + expect(page.rows.every((row) => row.traceId === null && row.spanId === null)).toBe(true); + const series = await loadErrorSeries(client, { ...options(logDataset), issue: escapedIssue }); + expect(series.reduce((sum, bucket) => sum + bucket.occurrences, 0)).toBe(3); + const missing = await loadErrorOccurrences(client, { + ...options(logDataset), + issue: { service: null, errorType: null, message: 'missing service and type' }, + }); + expect(missing.rows).toHaveLength(1); + }); + + it('counts each erroneous span once across event fan-out and exporter retries', async () => { + expect(schemas.get(traceDataset)?.source).toBe('traces'); + expect(await loadErrorSummary(client, options(traceDataset))).toMatchObject({ + occurrences: 2, + issues: 1, + }); + const issues = await loadErrorIssues(client, options(traceDataset)); + expect(issues.rows).toHaveLength(1); + expect(issues.rows[0]).toMatchObject({ ...escapedIssue, occurrences: 2 }); + const occurrences = await loadErrorOccurrences(client, { + ...options(traceDataset), + issue: escapedIssue, + }); + expect(occurrences.rows.map((row) => row.spanId).sort()).toEqual([spanId(1), spanId(2)]); + expect(occurrences.rows.every((row) => row.traceId === traceId)).toBe(true); + const series = await loadErrorSeries(client, options(traceDataset)); + expect(series.reduce((sum, bucket) => sum + bucket.occurrences, 0)).toBe(2); + }); + + it('executes the SQL investigation handoff and returns the original matching log fields', async () => { + const records = await query({ + sql: buildErrorExplorerSql({ ...options(logDataset), issue: escapedIssue }), + ...apmTransportBounds(bounds), + }); + expect(records).toHaveLength(3); + for (const record of records) { + expect(record).toMatchObject({ + 'service.name': escapedService, + 'exception.type': escapedType, + message: escapedMessage, + }); + expect(record.p_timestamp).toEqual(expect.any(String)); + expect(record).toHaveProperty('level'); + } + }); + + it('recognizes all OTLP error and fatal severity numbers from 17 through 24', async () => { + expect(schemas.get(numericDataset)?.source).toBe('logs'); + expect(await loadErrorSummary(client, options(numericDataset))).toMatchObject({ + occurrences: 8, + issues: 8, + }); + const issues = await loadErrorIssues(client, options(numericDataset)); + expect(issues.rows.map((issue) => issue.message).sort()).toEqual( + Array.from({ length: 8 }, (_, index) => `OTLP severity ${17 + index}`), + ); + }); + + it('excludes OTLP body-inferred categories from counts, frequency and SQL investigation', async () => { + const target = options(numericDataset); + // Verify the actual ingestion fallback, so this test cannot pass merely + // because the misleading records were not ingested. + const stored = await query({ + ...apmTransportBounds(bounds), + sql: `SELECT body, p_log_category FROM "${numericDataset}" WHERE severity_number = 0 ORDER BY body`, + }); + expect(stored).toEqual([ + { body: '0 errors found', p_log_category: 'ERROR' }, + { body: 'no fatal failures', p_log_category: 'FATAL' }, + ]); + expect(await loadErrorSummary(client, target)).toMatchObject({ occurrences: 8, issues: 8 }); + const series = await loadErrorSeries(client, target); + expect(series.reduce((sum, bucket) => sum + bucket.occurrences, 0)).toBe(8); + for (const { body } of stored) { + const issue = { service: 'severity-numbers', errorType: null, message: String(body) }; + expect((await loadErrorOccurrences(client, { ...target, issue })).rows).toEqual([]); + expect( + await query({ + sql: buildErrorExplorerSql({ ...target, issue }), + ...apmTransportBounds(bounds), + }), + ).toEqual([]); + } + }); + + it('keeps a span exception type and message paired across several exception events and a retry', async () => { + expect(await loadErrorSummary(client, options(pairDataset))).toMatchObject({ + occurrences: 1, + issues: 1, + }); + const issues = await loadErrorIssues(client, options(pairDataset)); + expect(issues.rows).toEqual([ + expect.objectContaining({ + service: escapedService, + errorType: 'A', + message: 'zeta', + occurrences: 1, + }), + ]); + const occurrences = await loadErrorOccurrences(client, { + ...options(pairDataset), + issue: issues.rows[0], + }); + expect(occurrences.rows.map((row) => row.spanId)).toEqual([spanId(9)]); + }); + + it('recognizes pino/bunyan numeric error and fatal levels exactly', async () => { + expect(schemas.get(levelDataset)?.severity.text).toContain('level'); + expect(await loadErrorSummary(client, options(levelDataset))).toMatchObject({ + occurrences: 2, + issues: 2, + }); + const issues = await loadErrorIssues(client, options(levelDataset)); + expect(issues.rows.map((issue) => issue.message).sort()).toEqual([ + 'pino level 50', + 'pino level 60', + ]); + }); + + it('honors an exclusive fractional upper bound and returns empty results before fixture ingestion', async () => { + const before = { + startTime: new Date(Date.parse(bounds.startTime) - 1).toISOString(), + endTime: bounds.startTime, + }; + const request = { ...options(logDataset), bounds: before }; + expect(await loadErrorSummary(client, request)).toMatchObject({ occurrences: 0, issues: 0 }); + expect((await loadErrorIssues(client, request)).rows).toEqual([]); + expect( + (await loadErrorSeries(client, request)).every((bucket) => bucket.occurrences === 0), + ).toBe(true); + expect((await loadErrorOccurrences(client, { ...request, issue: escapedIssue })).rows).toEqual( + [], + ); + }); + + afterAll(async () => { + const failures: string[] = []; + for (const dataset of attemptedDatasets) { + try { + const response = await fetch(`${origin}/api/v1/logstream/${dataset}`, { + method: 'DELETE', + headers, + signal: AbortSignal.timeout(10_000), + }); + if (!response.ok && response.status !== 404) + failures.push(`${dataset}: ${response.status} ${await response.text()}`); + } catch (error) { + failures.push(`${dataset}: ${String(error)}`); + } + } + expect(failures, 'Disposable error fixture datasets must be removed').toEqual([]); + }, 30_000); +}); diff --git a/frontend/src/lib/errorAnalytics.test.ts b/frontend/src/lib/errorAnalytics.test.ts new file mode 100644 index 000000000..254fdf3f2 --- /dev/null +++ b/frontend/src/lib/errorAnalytics.test.ts @@ -0,0 +1,792 @@ +import { describe, expect, it } from 'vitest'; +import { + buildErrorExplorerSql, + buildErrorQuery, + decodeErrorIssue, + decodeErrorSeries, + decodeErrorSummary, + discoverErrorDatasets, + errorIssueId, + inspectErrorSchema, + loadErrorIssues, + loadErrorOccurrences, + loadErrorOverview, + loadErrorSchema, + loadErrorSeries, + loadErrorSummary, + parseErrorIssueId, + UnsupportedErrorSchema, + ERROR_BUCKETS, + type ErrorIssueKey, + type ErrorQueryKind, + type ErrorSchema, +} from './errorAnalytics'; +import { handleErrorDemoQuery } from './errorAnalyticsDemo'; +import { createDemoClient } from './demo'; +import { ApiError } from './apiError'; +import { apmTransportBounds } from './apm'; +import type { DatasetInfo, LogRecord, ParseableClient } from './types'; + +const bounds = { startTime: '2026-10-12T15:00:12.500Z', endTime: '2026-10-12T16:00:34.250Z' }; +const traceInfo: DatasetInfo = { name: 'spans', logSourceFormats: ['otel-traces'] }; +const logInfo: DatasetInfo = { name: 'app', logSourceFormats: ['json'] }; +const traceFields = [ + 'p_timestamp', + 'service.name', + 'span_trace_id', + 'span_span_id', + 'span_status_code', + 'span_status_message', + 'span_name', + 'event_name', + 'event_exception.type', + 'event_exception.message', + 'exception.type', + 'exception.message', +]; +const traceSchema = inspectErrorSchema(traceFields, traceInfo); +const logSchema = inspectErrorSchema( + ['p_timestamp', 'level', 'severity_number', 'service', 'message', 'body', 'trace_id'], + logInfo, +); +const key: ErrorIssueKey = { service: 'checkout', errorType: null, message: 'boom' }; + +describe('error schema discovery', () => { + it('splits gated OTLP event exception fields from direct span attributes', () => { + expect(traceSchema).toMatchObject({ + source: 'traces', + timeColumn: 'p_timestamp', + service: ['service.name'], + errorType: ['exception.type'], + message: ['exception.message', 'span_status_message'], + exception: { + gate: 'event_name', + type: ['event_exception.type'], + message: ['event_exception.message'], + }, + traceId: 'span_trace_id', + spanId: 'span_span_id', + operation: 'span_name', + }); + }); + it('discovers log severity, message, service and evidence aliases in priority order', () => { + expect(logSchema).toMatchObject({ + source: 'logs', + service: ['service'], + errorType: [], + message: ['message', 'body'], + traceId: 'trace_id', + spanId: null, + severity: { text: ['level'], number: ['severity_number'] }, + }); + }); + it('explains ineligible datasets instead of fabricating empty analytics', () => { + const unsupported = (fields: string[], info: DatasetInfo) => { + try { + inspectErrorSchema(fields, info); + } catch (error) { + expect(error).toBeInstanceOf(UnsupportedErrorSchema); + return (error as Error).message; + } + throw new Error('expected unsupported'); + }; + expect(unsupported([], logInfo)).toMatch(/p_timestamp/); + expect(unsupported(['p_timestamp', 'message'], logInfo)).toMatch(/no severity field/); + expect(unsupported(['p_timestamp', 'service.name'], traceInfo)).toMatch( + /span_trace_id, span_span_id, span_status_code/, + ); + expect(unsupported(['p_timestamp', 'level'], { ...logInfo, telemetryType: 'metrics' })).toMatch( + /Metrics/, + ); + // A custom time partition is authoritative; never fall back to p_timestamp. + expect( + unsupported(['p_timestamp', 'level'], { ...logInfo, timePartition: 'event_time' }), + ).toMatch(/event_time is missing/); + expect( + inspectErrorSchema(['event_time', 'level'], { ...logInfo, timePartition: 'event_time' }) + .timeColumn, + ).toBe('event_time'); + }); + it('does not treat a derived log category as an explicit severity field', () => { + expect(() => inspectErrorSchema(['p_timestamp', 'p_log_category', 'body'], logInfo)).toThrow( + /no severity field/, + ); + }); +}); + +describe('error SQL contract', () => { + const build = (kind: ErrorQueryKind, extra: object = {}, schema = traceSchema) => + buildErrorQuery({ + dataset: 'spans', + schema, + bounds, + kind, + ...(kind === 'occurrences' ? { issue: key } : {}), + ...extra, + }); + it('deduplicates spans before aggregating and enforces exact bounds inside minute transport bounds', () => { + for (const kind of ['summary', 'series', 'issues', 'occurrences'] as const) { + const request = build(kind); + expect(request.startTime).toBe('2026-10-12T15:00:00.000Z'); + expect(request.endTime).toBe('2026-10-12T16:01:00.000Z'); + expect(request.sql).toMatch(/^\/\* parseable-errors:/); + expect(request.sql).toContain(`"p_timestamp" >= TIMESTAMP '2026-10-12T15:00:12.500Z'`); + expect(request.sql).toContain(`"p_timestamp" < TIMESTAMP '2026-10-12T16:00:34.250Z'`); + expect(request.sql).toContain( + `CASE WHEN TRY_CAST("span_status_code" AS DOUBLE) = 2 THEN 1 ELSE 0 END AS is_error`, + ); + expect(request.sql).toContain( + 'MAX(is_error) OVER (PARTITION BY service, trace_id, span_id) AS is_error', + ); + expect(request.sql).toContain('WHERE representative = 1 AND is_error = 1'); + expect(request.sql).not.toMatch(/LIMIT \d+\s*\)/); // no sampling inside the CTE + } + }); + it('gates only event fields and takes type and message together from one representative row per span', () => { + const { sql } = build('summary'); + expect(sql).toContain( + `CASE WHEN "event_name" = 'exception' THEN NULLIF(CAST("event_exception.type" AS VARCHAR), '') END AS exception_type`, + ); + expect(sql).toContain( + `COALESCE(NULLIF(CAST("exception.message" AS VARCHAR), ''), NULLIF(CAST("span_status_message" AS VARCHAR), '')) AS direct_message`, + ); + expect(sql).toContain( + 'ROW_NUMBER() OVER (PARTITION BY service, trace_id, span_id ORDER BY is_exception DESC, error_type ASC NULLS LAST, message ASC NULLS LAST) AS representative', + ); + // Independent per-field MAX would pair one event's type with another's message. + expect(sql).not.toMatch(/MAX\((error_type|message|exception_\w+|direct_\w+)\)/); + }); + it('classifies log errors by any severity field without dedup', () => { + const { sql } = build('summary', {}, logSchema); + expect(sql).toContain(`UPPER(TRIM(CAST("level" AS VARCHAR))) IN ('ERROR', 'ERROR2'`); + expect(sql).toContain(`OR TRY_CAST("level" AS DOUBLE) IN (50, 60)`); + expect(sql).toContain( + `OR TRY_CAST("severity_number" AS DOUBLE) IN (17, 18, 19, 20, 21, 22, 23, 24)`, + ); + // Numeric level codes are read from `level` only, never other text aliases. + const severity = inspectErrorSchema(['p_timestamp', 'severity', 'message'], logInfo); + expect(build('summary', {}, severity).sql).not.toContain('IN (50, 60)'); + expect(sql).not.toContain('GROUP BY "service.name"'); + }); + it('orders issue pages with stable tie-breakers and asks for one extra row', () => { + const orders = { + frequency: '"occurrences" DESC, "lastSeen" DESC', + latest: '"lastSeen" DESC, "occurrences" DESC', + first: '"firstSeen" DESC, "occurrences" DESC', + }; + for (const [sort, order] of Object.entries(orders)) + expect(build('issues', { sort, offset: 50 }).sql).toContain( + `ORDER BY ${order}, service ASC NULLS FIRST, error_type ASC NULLS FIRST, message ASC NULLS FIRST LIMIT 26 OFFSET 50`, + ); + expect(build('occurrences', { limit: 10 }).sql).toContain( + 'ORDER BY at DESC, trace_id ASC NULLS LAST, span_id ASC NULLS LAST, operation ASC NULLS LAST LIMIT 11 OFFSET 0', + ); + }); + it('excludes body-inferred categories from every log query, including SQL investigation', () => { + const schema = inspectErrorSchema( + ['p_timestamp', 'p_log_category', 'severity_number', 'severity_text', 'body'], + { name: 'logs', logSourceFormats: ['otel-logs'] }, + ); + for (const kind of ['summary', 'series', 'issues', 'occurrences', 'explore'] as const) { + const { sql } = buildErrorQuery({ + dataset: 'logs', + schema, + bounds, + kind, + ...(['occurrences', 'explore'].includes(kind) ? { issue: key } : {}), + }); + expect(sql).not.toContain('p_log_category'); + expect(sql).toContain( + 'TRY_CAST("severity_number" AS DOUBLE) IN (17, 18, 19, 20, 21, 22, 23, 24)', + ); + expect(sql).toContain('UPPER(TRIM(CAST("severity_text" AS VARCHAR)))'); + } + }); + it('uses at most 48 full-range buckets', () => { + expect(build('series').sql).toContain( + `DATE_BIN(INTERVAL '75454 milliseconds', at, TIMESTAMP '2026-10-12T15:00:12.500Z')`, + ); + expect(decodeErrorSeries([], bounds)).toHaveLength(ERROR_BUCKETS); + }); + it('keeps NULL identity distinct from literal placeholders in the issue predicate', () => { + const { sql } = build('occurrences', { + issue: { service: 'checkout', errorType: null, message: '(none)' }, + }); + expect(sql).toContain( + `WHERE service = 'checkout' AND error_type IS NULL AND message = '(none)'`, + ); + expect(parseErrorIssueId(errorIssueId(key))).toEqual(key); + expect(errorIssueId({ ...key, errorType: 'null' })).not.toBe(errorIssueId(key)); + expect(parseErrorIssueId('["a",1,null]')).toBeUndefined(); + expect(parseErrorIssueId('nope')).toBeUndefined(); + }); + it('quotes hostile identifiers and literals', () => { + const schema: ErrorSchema = { + ...logSchema, + service: [`svc"; DROP TABLE x; --`], + timeColumn: 'p_timestamp', + }; + const request = buildErrorQuery({ + dataset: `logs" WHERE 1=1 --`, + schema, + bounds, + kind: 'occurrences', + issue: { service: `it's'; --`, errorType: null, message: null }, + }); + expect(request.sql).toContain(`FROM "logs"" WHERE 1=1 --"`); + expect(request.sql).toContain(`CAST("svc""; DROP TABLE x; --" AS VARCHAR)`); + expect(request.sql).toContain(`service = 'it''s''; --'`); + expect(() => + buildErrorQuery({ + dataset: 'a', + schema, + bounds, + kind: 'occurrences', + issue: { ...key, message: 'x\0' }, + }), + ).toThrow(/null characters/); + expect(() => + buildErrorQuery({ + dataset: 'a', + schema: { ...schema, timeColumn: 'a\0' }, + bounds, + kind: 'summary', + }), + ).toThrow(); + }); + it('rejects invalid options', () => { + expect(() => build('occurrences', { issue: undefined })).toThrow(/Select an error issue/); + expect(() => build('issues', { limit: 26 })).toThrow(/between 1 and 25/); + expect(() => build('issues', { offset: -1 })).toThrow(/nonnegative/); + expect(() => build('issues', { sort: 'random' })).toThrow(/sort/); + expect(() => build('summary', { offset: 0 })).toThrow(/Pagination/); + expect(() => build('issues', { issue: key })).toThrow(/issue filter/); + expect(() => build('summary', { issue: { ...key, message: '' } })).toThrow(); + expect(() => + buildErrorQuery({ dataset: 'a', schema: traceSchema, kind: 'summary' } as never), + ).toThrow(/time range/); + expect(() => + buildErrorQuery({ + dataset: 'a', + schema: traceSchema, + ...bounds, + startTime: bounds.endTime, + kind: 'summary', + }), + ).toThrow(/Invalid time range/); + }); + it('builds marked raw-record explorer SQL with the same predicates as the aggregates', () => { + const sql = buildErrorExplorerSql({ dataset: 'app', schema: logSchema, bounds, issue: key }); + expect(sql).toMatch(/^\/\* parseable-errors:/); + expect(sql).toBe( + buildErrorQuery({ dataset: 'app', schema: logSchema, bounds, kind: 'explore', issue: key }) + .sql, + ); + expect(sql).toContain( + `SELECT * FROM "app" WHERE "p_timestamp" >= TIMESTAMP '2026-10-12T15:00:12.500Z' AND "p_timestamp" < TIMESTAMP '2026-10-12T16:00:34.250Z' AND (UPPER(TRIM(CAST("level" AS VARCHAR))) IN (`, + ); + expect(sql).toContain( + ` AND NULLIF(CAST("service" AS VARCHAR), '') = 'checkout' AND CAST(NULL AS VARCHAR) IS NULL AND COALESCE(NULLIF(CAST("message" AS VARCHAR), ''), NULLIF(CAST("body" AS VARCHAR), '')) = 'boom' ORDER BY "p_timestamp" DESC LIMIT 100`, + ); + const target = { dataset: 'app', schema: logSchema, bounds, issue: key }; + expect(buildErrorExplorerSql(target, 10_000)).toMatch(/LIMIT 10000$/); + expect(() => buildErrorExplorerSql(target, 10_001)).toThrow(/between 1 and 10000/); + expect(() => buildErrorExplorerSql(target, 0)).toThrow(/between 1 and 10000/); + expect(() => buildErrorQuery({ ...target, kind: 'explore', issue: undefined })).toThrow( + /Select an error issue/, + ); + expect(() => buildErrorQuery({ ...target, kind: 'explore', offset: 0 })).toThrow(/Pagination/); + expect(() => buildErrorExplorerSql({ ...target, schema: traceSchema })).toThrow(/log datasets/); + }); +}); + +describe('error result decoding', () => { + it('decodes summaries strictly', () => { + expect( + decodeErrorSummary([{ occurrences: 0, issues: 0, firstSeen: null, lastSeen: null }]), + ).toEqual({ occurrences: 0, issues: 0, firstSeen: null, lastSeen: null }); + expect( + decodeErrorSummary([ + { + occurrences: '3', + issues: 2, + firstSeen: '2026-10-12T15:01:00', + lastSeen: '2026-10-12T15:02:00', + }, + ]), + ).toMatchObject({ occurrences: 3, issues: 2 }); + for (const malformed of [ + {}, + [], + [{ occurrences: 1, issues: 2, firstSeen: 't', lastSeen: 't' }], + [{ occurrences: -1, issues: 0 }], + [{ occurrences: 1.5, issues: 1 }], + [{ occurrences: 2, issues: 0 }], + [{ occurrences: 1, issues: 1, firstSeen: 'yesterday', lastSeen: '2026-10-12T15:02:00Z' }], + [ + { + occurrences: 1, + issues: 1, + firstSeen: '2026-10-12T16:00:00Z', + lastSeen: '2026-10-12T15:00:00Z', + }, + ], + [{ occurrences: 0, issues: 0, firstSeen: '2026-10-12T15:00:00Z', lastSeen: null }], + ]) + expect(() => decodeErrorSummary(malformed)).toThrow(/Malformed error analytics result/); + }); + it('preserves null keys and rejects empty or non-string keys', () => { + const row = { + service: null, + errorType: null, + message: '(none)', + occurrences: 2, + firstSeen: '2026-10-12T15:01:00Z', + lastSeen: '2026-10-12T15:01:00Z', + }; + expect(decodeErrorIssue(row)).toMatchObject({ + service: null, + errorType: null, + message: '(none)', + }); + expect(() => decodeErrorIssue({ ...row, service: '' })).toThrow(/service/); + expect(() => decodeErrorIssue({ ...row, message: 5 })).toThrow(/message/); + expect(() => decodeErrorIssue({ ...row, occurrences: 0 })).toThrow(/occurrences/); + }); + it('fills series densely and rejects misaligned or duplicate buckets', () => { + const series = decodeErrorSeries( + [{ bucket: '2026-10-12T15:01:27.954', occurrences: 4 }], + bounds, + ); + expect(series[1]).toEqual({ bucket: '2026-10-12T15:01:27.954Z', occurrences: 4 }); + expect(series.reduce((sum, item) => sum + item.occurrences, 0)).toBe(4); + expect(() => + decodeErrorSeries([{ bucket: '2026-10-12T15:01:00Z', occurrences: 1 }], bounds), + ).toThrow(/bucket/); + expect(() => + decodeErrorSeries( + [ + { bucket: bounds.startTime, occurrences: 1 }, + { bucket: bounds.startTime, occurrences: 1 }, + ], + bounds, + ), + ).toThrow(/bucket/); + }); + it('rejects oversize and duplicate issue pages', async () => { + const row = { + service: 'a', + errorType: null, + message: null, + occurrences: 1, + firstSeen: bounds.startTime, + lastSeen: bounds.startTime, + }; + const client = (rows: unknown) => ({ query: async () => rows as LogRecord[] }); + const target = { dataset: 'app', schema: logSchema, bounds, limit: 1 }; + await expect(loadErrorIssues(client([row, row, row]), target)).rejects.toThrow(/page/); + await expect(loadErrorIssues(client([row, row]), { ...target, limit: 2 })).rejects.toThrow( + /identity/, + ); + await expect( + loadErrorIssues(client([row, { ...row, service: 'b' }]), target), + ).resolves.toMatchObject({ + rows: [{ service: 'a' }], + hasMore: true, + }); + await expect(loadErrorIssues(client('nope'), target)).rejects.toThrow(/rows/); + }); +}); + +describe('error dataset discovery', () => { + const fields: Record = { + logs: ['p_timestamp', 'level', 'message'], + traces: traceFields, + plain: ['p_timestamp', 'message'], + metrics: ['p_timestamp', 'level'], + }; + const client = (failures: Record = {}) => ({ + listDatasets: async () => + ['traces', 'logs', 'plain', 'metrics', 'denied', 'flaky', 'gone'].map((name) => ({ + name, + type: 'logs' as const, + })), + datasetInfo: async (name: string): Promise => { + if (name in failures) throw failures[name]; + return { + name, + logSourceFormats: [name === 'traces' ? 'otel-traces' : 'json'], + telemetryType: name === 'metrics' ? 'metrics' : 'logs', + }; + }, + schema: async (name: string) => fields[name] ?? [], + }); + it('trusts dataset info over listed type and separates transient failures', async () => { + const result = await discoverErrorDatasets( + client({ + denied: new ApiError('no', 403), + gone: new ApiError('no', 404), + flaky: new Error('Network unavailable'), + }), + ); + expect(result.datasets.map((item) => [item.name, item.source])).toEqual([ + ['logs', 'logs'], + ['traces', 'traces'], + ]); + expect(result.unchecked).toEqual(['flaky']); + }); + it('propagates authentication failures and aborts', async () => { + await expect( + discoverErrorDatasets(client({ flaky: new ApiError('login', 401) })), + ).rejects.toThrow('login'); + const controller = new AbortController(); + controller.abort(); + await expect(discoverErrorDatasets(client(), controller.signal)).rejects.toThrow(); + }); +}); + +describe('demo error analytics', () => { + const demo = createDemoClient(); + const now = Date.now(); + const range = { + startTime: new Date(now - 3_600_000).toISOString(), + endTime: new Date(now + 60_000).toISOString(), + }; + it('deduplicates repeated trace span rows before counting', async () => { + const schema = await loadErrorSchema(demo, 'demo_traces'); + const target = { dataset: 'demo_traces', schema, bounds: range }; + const { summary, issues, series } = await loadErrorOverview(demo, target); + expect(summary).toMatchObject({ occurrences: 3, issues: 1 }); + expect(issues.rows).toEqual([ + expect.objectContaining({ + service: 'checkout', + errorType: null, + message: 'Payment provider timed out', + occurrences: 3, + }), + ]); + expect(series.reduce((sum, item) => sum + item.occurrences, 0)).toBe(3); + const occurrences = await loadErrorOccurrences(demo, { ...target, issue: issues.rows[0] }); + expect(occurrences.rows).toHaveLength(3); + expect(new Set(occurrences.rows.map((item) => `${item.traceId}/${item.spanId}`)).size).toBe(3); + expect(occurrences.rows.every((item) => item.operation === 'create-order')).toBe(true); + }); + it('aggregates log errors from shared records with consistent totals and pages', async () => { + const schema = await loadErrorSchema(demo, 'application_logs'); + const target = { dataset: 'application_logs', schema, bounds: range }; + const summary = await loadErrorSummary(demo, target); + expect(summary.occurrences).toBe(8); // every 13th of 100 records is ERROR + const all = []; + for (let offset = 0; ; offset += 2) { + const page = await loadErrorIssues(demo, { ...target, limit: 2, offset, sort: 'latest' }); + all.push(...page.rows); + if (!page.hasMore) break; + } + expect(all).toHaveLength(summary.issues); + expect(all.reduce((sum, issue) => sum + issue.occurrences, 0)).toBe(summary.occurrences); + const lastSeen = all.map((issue) => Date.parse(issue.lastSeen)); + expect(lastSeen).toEqual([...lastSeen].sort((a, b) => b - a)); + for (const issue of all) { + const series = await loadErrorSeries(demo, { ...target, issue }); + expect(series.reduce((sum, item) => sum + item.occurrences, 0)).toBe(issue.occurrences); + const page = await loadErrorOccurrences(demo, { ...target, issue }); + expect(page.rows).toHaveLength(issue.occurrences); + expect(page.rows.every((item) => item.spanId === null && item.traceId)).toBe(true); + } + }); + it('respects exact range and rejects non-canonical SQL, unknown fields and datasets', async () => { + const schema = await loadErrorSchema(demo, 'application_logs'); + const empty = { + startTime: new Date(now + 3_600_000).toISOString(), + endTime: new Date(now + 7_200_000).toISOString(), + }; + await expect( + loadErrorSummary(demo, { dataset: 'application_logs', schema, bounds: empty }), + ).resolves.toEqual({ + occurrences: 0, + issues: 0, + firstSeen: null, + lastSeen: null, + }); + const request = buildErrorQuery({ + dataset: 'application_logs', + schema, + bounds: range, + kind: 'summary', + }); + await expect(demo.query({ ...request, sql: `${request.sql} UNION SELECT 1` })).rejects.toThrow( + /Unsupported demo error query/, + ); + await expect( + demo.query( + buildErrorQuery({ + dataset: 'application_logs', + schema: { ...schema, errorType: ['nope'] }, + bounds: range, + kind: 'summary', + }), + ), + ).rejects.toThrow(/Unknown demo field/); + await expect( + demo.query(buildErrorQuery({ dataset: 'missing', schema, bounds: range, kind: 'summary' })), + ).rejects.toThrow(/Unknown demo dataset/); + }); + it('runs the explorer SQL for a log issue and rejects modified SQL', async () => { + const schema = await loadErrorSchema(demo, 'application_logs'); + const issue = { + service: 'api-gateway', + errorType: null, + message: 'Upstream request timed out after 3000ms', + }; + const sql = buildErrorExplorerSql({ + dataset: 'application_logs', + schema, + bounds: range, + issue, + }); + const request = { sql, ...apmTransportBounds(range) }; + const records = await demo.query(request); + // Fixture records 0 and 65 are the api-gateway ERROR rows. + expect(records).toHaveLength(2); + expect(records).toEqual([ + expect.objectContaining({ + level: 'ERROR', + service: 'api-gateway', + status_code: 504, + host: 'prod-node-01', + }), + expect.objectContaining({ + level: 'ERROR', + service: 'api-gateway', + status_code: 504, + host: 'prod-node-02', + }), + ]); + expect(Date.parse(String(records[0].p_timestamp))).toBeGreaterThan( + Date.parse(String(records[1].p_timestamp)), + ); + await expect( + demo.query({ ...request, sql: sql.replace('LIMIT 100', 'LIMIT 1000') }), + ).rejects.toThrow(/Unsupported demo error query/); + await expect( + demo.query({ ...request, sql: sql.replace(`= 'api-gateway'`, `<> 'api-gateway'`) }), + ).rejects.toThrow(/Unsupported demo error query/); + await expect(demo.query({ ...request, startTime: range.startTime })).rejects.toThrow( + /Unsupported demo error query/, + ); + }); + it('compares numeric status and severity exactly without truncation', () => { + const at = '2026-10-12T15:30:00.000Z'; + const logs: LogRecord[] = [17, '24', 24.9, '2.9e1', '18.0', 16.5, ' 17', 'x', null].map( + (severity_number, index) => ({ p_timestamp: at, severity_number, message: `m${index}` }), + ); + const numberSchema = inspectErrorSchema(['p_timestamp', 'severity_number', 'message'], logInfo); + const summary = handleErrorDemoQuery( + buildErrorQuery({ dataset: 'app', schema: numberSchema, bounds, kind: 'issues' }), + logs, + )!; + // The live engine accepts surrounding whitespace but rejects fractional codes. + expect(summary.map((row) => row.message).sort()).toEqual(['m0', 'm1', 'm4', 'm6']); + const spans: LogRecord[] = [2, '2', 2.9, '2.0', 1].map((span_status_code, index) => ({ + p_timestamp: at, + 'service.name': 'svc', + span_trace_id: 't', + span_span_id: `s${index}`, + span_status_code, + })); + const statusSchema = inspectErrorSchema( + ['p_timestamp', 'service.name', 'span_trace_id', 'span_span_id', 'span_status_code'], + traceInfo, + ); + expect( + handleErrorDemoQuery( + buildErrorQuery({ dataset: 'spans', schema: statusSchema, bounds, kind: 'summary' }), + spans, + ), + ).toEqual([expect.objectContaining({ occurrences: 3, issues: 1 })]); + }); + it('ignores body-inferred error and fatal categories but preserves explicit severities', () => { + // OTLP ingestion infers p_log_category from body substrings when severity is + // unspecified. These rows model the stored output, not the raw OTLP input. + const logs: LogRecord[] = [ + { + body: '0 errors found', + severity_number: 0, + severity_text: 'UNSPECIFIED', + p_log_category: 'ERROR', + }, + { + body: 'no fatal failures', + severity_number: 0, + severity_text: 'UNSPECIFIED', + p_log_category: 'FATAL', + }, + { body: 'numeric error', severity_number: 17, p_log_category: 'ERROR' }, + { body: 'numeric fatal', severity_number: 24, p_log_category: 'FATAL' }, + { body: 'text error', severity_text: 'ERROR' }, + { body: 'pino error', level: 50 }, + ].map((row) => ({ ...row, p_timestamp: '2026-10-12T15:30:00.000Z' })); + const schema = inspectErrorSchema( + ['p_timestamp', 'p_log_category', 'severity_number', 'severity_text', 'level', 'body'], + { name: 'logs', logSourceFormats: ['otel-logs'] }, + ); + const target = { dataset: 'logs', schema, bounds }; + expect(handleErrorDemoQuery(buildErrorQuery({ ...target, kind: 'summary' }), logs)).toEqual([ + expect.objectContaining({ occurrences: 4, issues: 4 }), + ]); + const issues = handleErrorDemoQuery(buildErrorQuery({ ...target, kind: 'issues' }), logs)!; + expect(issues.map((row) => row.message).sort()).toEqual([ + 'numeric error', + 'numeric fatal', + 'pino error', + 'text error', + ]); + expect( + handleErrorDemoQuery( + buildErrorQuery({ + ...target, + kind: 'explore', + issue: { service: null, errorType: null, message: '0 errors found' }, + }), + logs, + ), + ).toEqual([]); + }); + it('counts pino/bunyan numeric error and fatal levels exactly, alongside text levels', () => { + const at = '2026-10-12T15:30:00.000Z'; + const levels = [50, 60, '50', ' 60', 'error', 40, 30, 50.9, 59.5, 70, '50x', null]; + const logs: LogRecord[] = levels.map((level, index) => ({ + p_timestamp: at, + level, + msg: `m${index}`, + })); + const levelSchema = inspectErrorSchema(['p_timestamp', 'level', 'msg'], logInfo); + const issues = handleErrorDemoQuery( + buildErrorQuery({ dataset: 'app', schema: levelSchema, bounds, kind: 'issues' }), + logs, + )!; + // error (50), fatal (60) and the text alias; warn (40), info (30), fractional + // codes and values above fatal are not errors. + expect(issues.map((row) => row.message).sort()).toEqual(['m0', 'm1', 'm2', 'm3', 'm4']); + }); + it('keeps every span exception type and message paired when one span has several exception events', () => { + const at = '2026-10-12T15:30:00.000Z'; + const row = (extra: LogRecord): LogRecord => ({ + p_timestamp: at, + 'service.name': 'checkout', + span_trace_id: 't1', + span_span_id: 's1', + span_status_code: 2, + span_status_message: 'status says something else', + 'exception.type': 'DirectType', + span_name: 'pay', + event_name: null, + ...extra, + }); + const exception = (type: string, message: string) => + row({ + event_name: 'exception', + 'event_exception.type': type, + 'event_exception.message': message, + }); + // Two different exception events, the span row itself, and an exporter retry + // of all three. Per-field MAX would invent (B, zeta), which no event carried. + const once = [row({}), exception('A', 'zeta'), exception('B', 'alpha')]; + const run = (kind: ErrorQueryKind) => + handleErrorDemoQuery( + buildErrorQuery({ dataset: 'spans', schema: traceSchema, bounds, kind }), + [...once, ...once], + )!; + expect(run('summary')).toEqual([expect.objectContaining({ occurrences: 1, issues: 1 })]); + expect(run('issues')).toEqual([ + expect.objectContaining({ errorType: 'A', message: 'zeta', occurrences: 1 }), + ]); + // The pick does not depend on row order. + const reversed = handleErrorDemoQuery( + buildErrorQuery({ dataset: 'spans', schema: traceSchema, bounds, kind: 'issues' }), + [...once].reverse(), + )!; + expect(reversed).toEqual([expect.objectContaining({ errorType: 'A', message: 'zeta' })]); + // Without a populated exception event, the direct span metadata is the pair. + const direct = handleErrorDemoQuery( + buildErrorQuery({ dataset: 'spans', schema: traceSchema, bounds, kind: 'issues' }), + [row({}), row({ event_name: 'log', 'event_exception.type': 'Ignored' })], + )!; + expect(direct).toEqual([ + expect.objectContaining({ + errorType: 'DirectType', + message: 'status says something else', + }), + ]); + }); + it('groups an event-only exception and a direct span attribute with the same key', () => { + const at = '2026-10-12T15:30:00.000Z'; + const span = (id: string, extra: LogRecord): LogRecord => ({ + p_timestamp: at, + 'service.name': 'checkout', + span_trace_id: `t${id}`, + span_span_id: `s${id}`, + span_status_code: 2, + span_status_message: '', + span_name: 'pay', + event_name: null, + ...extra, + }); + const rows = [ + span('1', {}), + span('1', { + event_name: 'exception', + 'event_exception.type': 'Timeout', + 'event_exception.message': 'gateway', + }), + span('1', { + event_name: 'log', + 'event_exception.type': 'Ignored', + 'event_exception.message': 'ignored', + }), + span('2', { 'exception.type': 'Timeout', 'exception.message': 'gateway' }), + span('3', { span_status_message: 'gateway' }), // NULL type stays its own issue + span('4', { + span_status_code: '2', + 'exception.type': '(none)', + 'exception.message': 'gateway', + }), + span('5', { + span_status_code: 1, + 'exception.type': 'Timeout', + 'exception.message': 'gateway', + }), + ]; + const run = (kind: ErrorQueryKind) => + handleErrorDemoQuery( + buildErrorQuery({ dataset: 'spans', schema: traceSchema, bounds, kind }), + rows, + )!; + expect(run('summary')).toEqual([expect.objectContaining({ occurrences: 4, issues: 3 })]); + expect(run('issues')).toEqual([ + expect.objectContaining({ + service: 'checkout', + errorType: 'Timeout', + message: 'gateway', + occurrences: 2, + }), + expect.objectContaining({ errorType: null, message: 'gateway', occurrences: 1 }), + expect.objectContaining({ errorType: '(none)', message: 'gateway', occurrences: 1 }), + ]); + }); +}); + +describe('loader wiring', () => { + it('passes the signal and flat bounds through', async () => { + const seen: AbortSignal[] = []; + const signal = new AbortController().signal; + const client: Pick = { + query: async (_request, s) => { + seen.push(s!); + return [{ occurrences: 0, issues: 0, firstSeen: null, lastSeen: null }]; + }, + }; + await loadErrorSummary(client, { dataset: 'app', schema: logSchema, ...bounds }, signal); + expect(seen).toEqual([signal]); + }); +}); diff --git a/frontend/src/lib/errorAnalytics.ts b/frontend/src/lib/errorAnalytics.ts new file mode 100644 index 000000000..41e35f4d4 --- /dev/null +++ b/frontend/src/lib/errorAnalytics.ts @@ -0,0 +1,653 @@ +import { parseEventTimestamp } from '../components/explorer/timestamp'; +import { ApiError } from './apiError'; +import { apmTransportBounds } from './apm'; +import { quoteIdentifier, quoteLiteral, timeBounds } from './query'; +import type { DatasetInfo, LogRecord, ParseableClient, QueryRequest } from './types'; + +export type ErrorSource = 'traces' | 'logs'; +/** Half-open ingestion-time range: start inclusive, end exclusive. */ +export type ErrorBounds = { startTime: string; endTime: string }; +/** Discovered columns, each list in fixed priority order and coalesced per row. */ +export type ErrorSchema = { + source: ErrorSource; + timeColumn: string; + service: string[]; + errorType: string[]; + message: string[]; + /** Traces only: event_exception.* fields are read from exception event rows + * (when `gate` names the event column). A span whose exception row is populated + * takes its type and message as a pair from that row; the direct, ungated + * errorType/message attributes are used only when no such row exists. */ + exception: { gate: string | null; type: string[]; message: string[] }; + traceId: string | null; + spanId: string | null; + operation: string | null; + /** Logs only. A row is an error when any field matches; a `level` text field + * also matches the exact numeric codes in ERROR_LEVEL_NUMBER. */ + severity: { text: string[]; number: string[] }; +}; +/** Exact group identity. null means absent, NULL or empty, never a placeholder. */ +export type ErrorIssueKey = { + service: string | null; + errorType: string | null; + message: string | null; +}; +export type ErrorIssue = ErrorIssueKey & { + id: string; + occurrences: number; + firstSeen: string; + lastSeen: string; +}; +export type ErrorSummary = { + occurrences: number; + issues: number; + firstSeen: string | null; + lastSeen: string | null; +}; +export type ErrorBucket = { bucket: string; occurrences: number }; +export type ErrorOccurrence = { + time: string; + traceId: string | null; + spanId: string | null; + operation: string | null; +}; +/** frequency: most occurrences; latest: most recently seen; first: newest issues first. */ +export type ErrorSort = 'frequency' | 'latest' | 'first'; +/** explore: raw stored log records of one issue, for the SQL explorer. */ +export type ErrorQueryKind = 'summary' | 'series' | 'issues' | 'occurrences' | 'explore'; +export type ErrorPage = { rows: T[]; offset: number; hasMore: boolean }; +export type ErrorTarget = Partial & { + dataset: string; + schema: ErrorSchema; + bounds?: ErrorBounds; +}; +export type ErrorQueryOptions = ErrorTarget & { + kind: ErrorQueryKind; + issue?: ErrorIssueKey; + sort?: ErrorSort; + offset?: number; + limit?: number; +}; +/** Canonical demo marker payload: flat normalized bounds, defaults applied. */ +export type ErrorQueryMeta = ErrorBounds & { + dataset: string; + schema: ErrorSchema; + kind: ErrorQueryKind; + issue?: ErrorIssueKey; + sort?: ErrorSort; + offset?: number; + limit?: number; +}; + +export const ERROR_PAGE_SIZE = 25; +export const ERROR_BUCKETS = 48; +export const ERROR_SORTS: readonly ErrorSort[] = ['frequency', 'latest', 'first']; +export const ERROR_QUERY_MARKER = 'parseable-errors'; +export const ERROR_SEVERITY_TEXT = [ + 'ERROR', + 'ERROR2', + 'ERROR3', + 'ERROR4', + 'ERR', + 'FATAL', + 'FATAL2', + 'FATAL3', + 'FATAL4', + 'CRITICAL', + 'CRIT', + 'ALERT', + 'EMERG', + 'EMERGENCY', + 'PANIC', + 'SEVERE', +] as const; +/** OTLP SeverityNumber ERROR..FATAL4. */ +export const ERROR_SEVERITY_NUMBER = [17, 18, 19, 20, 21, 22, 23, 24] as const; +/** pino/bunyan numeric levels error (50) and fatal (60), read from `level` only. */ +export const ERROR_LEVEL_NUMBER = [50, 60] as const; +export const ERROR_LEVEL_NUMBER_FIELD = 'level'; +export const ERROR_EXPLORE_LIMIT = { default: 100, max: 10_000 } as const; + +const ALIASES = { + // p_log_category can be inferred from body text when OTLP severity is unset + // (src/otel/logs.rs). Only explicit severity fields may classify an error. + severityText: ['level', 'log_level', 'severity', 'severity_text'], + severityNumber: ['severity_number'], + logService: ['service.name', 'service', 'service_name'], + logErrorType: ['exception.type', 'error.type', 'error.kind', 'exception_type', 'error_type'], + logMessage: ['message', 'msg', 'body', 'log', 'error.message'], + logTraceId: ['trace_id', 'traceId', 'span_trace_id'], + logSpanId: ['span_id', 'spanId', 'span_span_id'], + // Event fields exist only on expanded event rows; direct span attributes are ungated. + traceEventType: ['event_exception.type'], + traceEventMessage: ['event_exception.message'], + traceErrorType: ['exception.type'], + traceMessage: ['exception.message', 'span_status_message'], +}; +const TRACE_REQUIRED = ['service.name', 'span_trace_id', 'span_span_id', 'span_status_code']; + +/** The dataset is readable but cannot be analyzed; message explains why. */ +export class UnsupportedErrorSchema extends Error { + name = 'UnsupportedErrorSchema'; +} + +/** Throws UnsupportedErrorSchema explaining why the dataset cannot be analyzed. */ +export function inspectErrorSchema(fields: string[], info: DatasetInfo): ErrorSchema { + const has = new Set(fields); + const present = (names: string[]) => names.filter((name) => has.has(name)); + if (info.telemetryType === 'metrics') + throw new UnsupportedErrorSchema('Metrics datasets do not contain error events'); + // A time partition changes what the time range means; never substitute p_timestamp. + const timeColumn = info.timePartition || 'p_timestamp'; + if (!has.has(timeColumn)) + throw new UnsupportedErrorSchema( + info.timePartition + ? `This dataset's time partition column ${info.timePartition} is missing from its schema` + : 'This dataset does not have the p_timestamp column', + ); + if (info.logSourceFormats.includes('otel-traces')) { + const missing = TRACE_REQUIRED.filter((field) => !has.has(field)); + if (missing.length) + throw new UnsupportedErrorSchema( + `This dataset does not have the required OTLP span fields: ${missing.join(', ')}`, + ); + return { + source: 'traces', + timeColumn, + service: ['service.name'], + errorType: present(ALIASES.traceErrorType), + message: present(ALIASES.traceMessage), + exception: { + gate: has.has('event_name') ? 'event_name' : null, + type: present(ALIASES.traceEventType), + message: present(ALIASES.traceEventMessage), + }, + traceId: 'span_trace_id', + spanId: 'span_span_id', + operation: has.has('span_name') ? 'span_name' : null, + severity: { text: [], number: [] }, + }; + } + const severity = { + text: present(ALIASES.severityText), + number: present(ALIASES.severityNumber), + }; + if (!severity.text.length && !severity.number.length) + throw new UnsupportedErrorSchema( + `This dataset has no severity field (${[...ALIASES.severityText, ...ALIASES.severityNumber].join(', ')})`, + ); + return { + source: 'logs', + timeColumn, + service: present(ALIASES.logService), + errorType: present(ALIASES.logErrorType), + message: present(ALIASES.logMessage), + exception: { gate: null, type: [], message: [] }, + traceId: present(ALIASES.logTraceId)[0] ?? null, + spanId: present(ALIASES.logSpanId)[0] ?? null, + operation: null, + severity, + }; +} + +export async function loadErrorSchema( + client: Pick, + dataset: string, + signal?: AbortSignal, +) { + const [info, fields] = await Promise.all([ + client.datasetInfo(dataset, signal), + client.schema(dataset, signal), + ]); + return inspectErrorSchema(fields, info); +} + +export type ErrorDataset = { name: string; source: ErrorSource; schema: ErrorSchema }; +export async function discoverErrorDatasets( + client: Pick, + signal?: AbortSignal, +) { + // Listed types can be placeholders; dataset info decides eligibility. + const datasets = await client.listDatasets(signal); + const found: ErrorDataset[] = [], + unchecked: string[] = []; + let next = 0; + await Promise.all( + Array.from({ length: Math.min(4, datasets.length) }, async () => { + while (next < datasets.length) { + signal?.throwIfAborted(); + const dataset = datasets[next++]; + let schema: ErrorSchema; + try { + schema = await loadErrorSchema(client, dataset.name, signal); + } catch (error) { + if ( + signal?.aborted || + (error instanceof Error && error.name === 'AbortError') || + (error instanceof ApiError && error.status === 401) + ) + throw error; + if ( + !(error instanceof UnsupportedErrorSchema) && + !(error instanceof ApiError && (error.status === 403 || error.status === 404)) + ) + unchecked.push(dataset.name); + continue; + } + found.push({ name: dataset.name, source: schema.source, schema }); + } + }), + ); + return { + datasets: found.sort((a, b) => a.name.localeCompare(b.name)), + unchecked: unchecked.sort(), + }; +} + +export function errorBucketInterval(bounds: ErrorBounds) { + const normalized = timeBounds(bounds); + return Math.ceil( + (Date.parse(normalized.endTime) - Date.parse(normalized.startTime)) / ERROR_BUCKETS, + ); +} +/** URL-safe stable identity; JSON keeps null distinct from the string "null". */ +export function errorIssueId(key: ErrorIssueKey): string { + return JSON.stringify([key.service, key.errorType, key.message]); +} +export function parseErrorIssueId(id: string): ErrorIssueKey | undefined { + try { + const value: unknown = JSON.parse(id); + if ( + Array.isArray(value) && + value.length === 3 && + value.every((item) => item === null || typeof item === 'string') + ) + return { service: value[0], errorType: value[1], message: value[2] }; + } catch { + /* not an issue id */ + } + return undefined; +} + +function resolveBounds(target: ErrorTarget): ErrorBounds { + const value = target.bounds ?? { startTime: target.startTime!, endTime: target.endTime! }; + if (!value.startTime || !value.endTime) throw new Error('Error analytics needs a time range'); + return timeBounds(value); +} +function validateSchema(schema: ErrorSchema) { + const lists = [ + schema.service, + schema.errorType, + schema.message, + schema.exception?.type, + schema.exception?.message, + schema.severity?.text, + schema.severity?.number, + ]; + if ( + (schema.source !== 'traces' && schema.source !== 'logs') || + typeof schema.timeColumn !== 'string' || + lists.some((list) => !Array.isArray(list) || list.some((f) => typeof f !== 'string')) + ) + throw new Error('Invalid error schema'); + if (schema.source === 'logs' && !schema.severity.text.length && !schema.severity.number.length) + throw new Error('Invalid error schema'); + if (schema.source === 'traces' && (!schema.traceId || !schema.spanId)) + throw new Error('Invalid error schema'); +} +function validateKey(issue: ErrorIssueKey) { + for (const value of [issue.service, issue.errorType, issue.message]) + if (value !== null && (typeof value !== 'string' || value === '')) + throw new Error('Invalid error issue'); +} +const text = (field: string) => `NULLIF(CAST(${quoteIdentifier(field)} AS VARCHAR), '')`; +const coalesce = (fields: string[]) => + fields.length === 0 + ? 'CAST(NULL AS VARCHAR)' + : fields.length === 1 + ? text(fields[0]) + : `COALESCE(${fields.map(text).join(', ')})`; +const equals = (column: string, value: string | null) => + value === null ? `${column} IS NULL` : `${column} = ${quoteLiteral(value)}`; +export function errorIssuePredicate(issue: ErrorIssueKey, prefix = '') { + return [ + equals(`${prefix}service`, issue.service), + equals(`${prefix}error_type`, issue.errorType), + equals(`${prefix}message`, issue.message), + ].join(' AND '); +} +function logIssuePredicate(schema: ErrorSchema, issue: ErrorIssueKey) { + return [ + equals(coalesce(schema.service), issue.service), + equals(coalesce(schema.errorType), issue.errorType), + equals(coalesce(schema.message), issue.message), + ].join(' AND '); +} +function severityPredicate(schema: ErrorSchema) { + const values = ERROR_SEVERITY_TEXT.map((value) => quoteLiteral(value)).join(', '); + // Exact comparison: fractional codes such as 24.9 or 50.9 are not severities. + const exact = (field: string, codes: readonly number[]) => + `TRY_CAST(${quoteIdentifier(field)} AS DOUBLE) IN (${codes.join(', ')})`; + return `(${[ + ...schema.severity.text.map( + (field) => `UPPER(TRIM(CAST(${quoteIdentifier(field)} AS VARCHAR))) IN (${values})`, + ), + ...schema.severity.text + .filter((field) => field === ERROR_LEVEL_NUMBER_FIELD) + .map((field) => exact(field, ERROR_LEVEL_NUMBER)), + ...schema.severity.number.map((field) => exact(field, ERROR_SEVERITY_NUMBER)), + ].join(' OR ')})`; +} +function timePredicate(column: string, bounds: ErrorBounds) { + return `${column} >= TIMESTAMP ${quoteLiteral(bounds.startTime)} AND ${column} < TIMESTAMP ${quoteLiteral(bounds.endTime)}`; +} + +/** Normalized error rows: one per occurrence. Trace spans are deduplicated by + * service + trace + span before anything is counted, because event and link + * expansion repeats a span. Each span takes its type and message together from + * one representative row so the pair is never mixed across events: rows whose + * exception event fields are populated win; direct span attributes are used only + * when the span has no such row; ties break on the pair itself (ASC, NULLS LAST). */ +function errorsCte(dataset: string, schema: ErrorSchema, bounds: ErrorBounds) { + const time = quoteIdentifier(schema.timeColumn); + const table = quoteIdentifier(dataset); + if (schema.source === 'traces') { + const service = quoteIdentifier('service.name'), + trace = quoteIdentifier(schema.traceId!), + span = quoteIdentifier(schema.spanId!); + const event = (fields: string[]) => + fields.length === 0 + ? 'CAST(NULL AS VARCHAR)' + : schema.exception.gate + ? `CASE WHEN ${quoteIdentifier(schema.exception.gate)} = 'exception' THEN ${coalesce(fields)} END` + : coalesce(fields); + const partition = 'PARTITION BY service, trace_id, span_id'; + return `WITH span_rows AS ( + SELECT ${service} AS service, ${trace} AS trace_id, ${span} AS span_id, ${time} AS at, + CASE WHEN TRY_CAST(${quoteIdentifier('span_status_code')} AS DOUBLE) = 2 THEN 1 ELSE 0 END AS is_error, + ${event(schema.exception.type)} AS exception_type, ${event(schema.exception.message)} AS exception_message, + ${coalesce(schema.errorType)} AS direct_type, ${coalesce(schema.message)} AS direct_message, + ${schema.operation ? text(schema.operation) : 'CAST(NULL AS VARCHAR)'} AS operation + FROM ${table} + WHERE ${service} IS NOT NULL AND ${service} <> '' AND ${trace} IS NOT NULL AND ${trace} <> '' AND ${span} IS NOT NULL AND ${span} <> '' + AND ${timePredicate(time, bounds)} + ), candidates AS ( + SELECT service, trace_id, span_id, at, is_error, operation, + CASE WHEN exception_type IS NOT NULL OR exception_message IS NOT NULL THEN 1 ELSE 0 END AS is_exception, + CASE WHEN exception_type IS NOT NULL OR exception_message IS NOT NULL THEN exception_type ELSE direct_type END AS error_type, + CASE WHEN exception_type IS NOT NULL OR exception_message IS NOT NULL THEN exception_message ELSE direct_message END AS message + FROM span_rows + ), spans AS ( + SELECT service, trace_id, span_id, error_type, message, + MIN(at) OVER (${partition}) AS at, + MAX(is_error) OVER (${partition}) AS is_error, + MAX(operation) OVER (${partition}) AS operation, + ROW_NUMBER() OVER (${partition} ORDER BY is_exception DESC, error_type ASC NULLS LAST, message ASC NULLS LAST) AS representative + FROM candidates + ), errors AS ( + SELECT service, error_type, message, at, trace_id, span_id, operation FROM spans WHERE representative = 1 AND is_error = 1 + )`; + } + return `WITH errors AS ( + SELECT ${coalesce(schema.service)} AS service, ${coalesce(schema.errorType)} AS error_type, ${coalesce(schema.message)} AS message, ${time} AS at, + ${schema.traceId ? text(schema.traceId) : 'CAST(NULL AS VARCHAR)'} AS trace_id, ${schema.spanId ? text(schema.spanId) : 'CAST(NULL AS VARCHAR)'} AS span_id, CAST(NULL AS VARCHAR) AS operation + FROM ${table} + WHERE ${timePredicate(time, bounds)} AND ${severityPredicate(schema)} + )`; +} + +const TIE_BREAK = 'service ASC NULLS FIRST, error_type ASC NULLS FIRST, message ASC NULLS FIRST'; +const SORT_ORDER: Record = { + frequency: `"occurrences" DESC, "lastSeen" DESC, ${TIE_BREAK}`, + latest: `"lastSeen" DESC, "occurrences" DESC, ${TIE_BREAK}`, + first: `"firstSeen" DESC, "occurrences" DESC, ${TIE_BREAK}`, +}; + +function canonical(options: ErrorQueryOptions): ErrorQueryMeta { + const { dataset, schema, kind } = options; + if (!['summary', 'series', 'issues', 'occurrences', 'explore'].includes(kind)) + throw new Error('Unsupported error query kind'); + validateSchema(schema); + quoteIdentifier(dataset); + const meta: ErrorQueryMeta = { ...resolveBounds(options), dataset, schema, kind }; + if (options.issue) { + if (kind !== 'series' && kind !== 'occurrences' && kind !== 'explore') + throw new Error(`An issue filter is not supported for ${kind}`); + validateKey(options.issue); + meta.issue = { + service: options.issue.service, + errorType: options.issue.errorType, + message: options.issue.message, + }; + } else if (kind === 'occurrences' || kind === 'explore') throw new Error('Select an error issue'); + if (kind === 'explore' && schema.source !== 'logs') + throw new Error('Raw error records are explored for log datasets; open traces natively'); + if (kind === 'issues') { + const sort = options.sort ?? 'frequency'; + if (!ERROR_SORTS.includes(sort)) throw new Error('Unsupported error sort'); + meta.sort = sort; + } else if (options.sort !== undefined) throw new Error(`Sorting is not supported for ${kind}`); + if (kind === 'issues' || kind === 'occurrences') { + const limit = options.limit ?? ERROR_PAGE_SIZE, + offset = options.offset ?? 0; + if (!Number.isInteger(limit) || limit < 1 || limit > ERROR_PAGE_SIZE) + throw new Error(`Error page size must be between 1 and ${ERROR_PAGE_SIZE}`); + if (!Number.isSafeInteger(offset) || offset < 0) + throw new Error('Error page offset must be a nonnegative integer'); + meta.limit = limit; + meta.offset = offset; + } else if (kind === 'explore') { + const limit = options.limit ?? ERROR_EXPLORE_LIMIT.default; + if (!Number.isInteger(limit) || limit < 1 || limit > ERROR_EXPLORE_LIMIT.max) + throw new Error(`Limit must be between 1 and ${ERROR_EXPLORE_LIMIT.max}`); + if (options.offset !== undefined) throw new Error('Pagination is not supported for explore'); + meta.limit = limit; + } else if (options.limit !== undefined || options.offset !== undefined) + throw new Error(`Pagination is not supported for ${kind}`); + return meta; +} + +/** Full-range SQL over a normalized errors CTE; no sampling before aggregates. + * Transport bounds are expanded to whole minutes; SQL enforces the exact range. */ +export function buildErrorQuery(options: ErrorQueryOptions): QueryRequest { + const meta = canonical(options); + const { schema, kind } = meta; + const bounds = { startTime: meta.startTime, endTime: meta.endTime }; + if (kind === 'explore') { + const time = quoteIdentifier(schema.timeColumn); + const select = `SELECT * FROM ${quoteIdentifier(meta.dataset)} WHERE ${timePredicate(time, bounds)} AND ${severityPredicate(schema)} AND ${logIssuePredicate(schema, meta.issue!)} ORDER BY ${time} DESC LIMIT ${meta.limit}`; + return request(meta, bounds, select); + } + const cte = errorsCte(meta.dataset, schema, bounds); + const issueFilter = meta.issue ? ` WHERE ${errorIssuePredicate(meta.issue)}` : ''; + const page = () => `LIMIT ${meta.limit! + 1} OFFSET ${meta.offset}`; + let select: string; + switch (kind) { + case 'summary': + select = `SELECT COALESCE(SUM(n), 0) AS "occurrences", COUNT(*) AS "issues", MIN(first_seen) AS "firstSeen", MAX(last_seen) AS "lastSeen" FROM (SELECT COUNT(*) AS n, MIN(at) AS first_seen, MAX(at) AS last_seen FROM errors GROUP BY service, error_type, message) issues`; + break; + case 'series': + select = `SELECT DATE_BIN(INTERVAL '${errorBucketInterval(bounds)} milliseconds', at, TIMESTAMP ${quoteLiteral(bounds.startTime)}) AS "bucket", COUNT(*) AS "occurrences" FROM errors${issueFilter} GROUP BY "bucket" ORDER BY "bucket" ASC`; + break; + case 'issues': + select = `SELECT service AS "service", error_type AS "errorType", message AS "message", COUNT(*) AS "occurrences", MIN(at) AS "firstSeen", MAX(at) AS "lastSeen" FROM errors GROUP BY service, error_type, message ORDER BY ${SORT_ORDER[meta.sort!]} ${page()}`; + break; + case 'occurrences': + select = `SELECT at AS "time", trace_id AS "traceId", span_id AS "spanId", operation AS "operation" FROM errors${issueFilter} ORDER BY at DESC, trace_id ASC NULLS LAST, span_id ASC NULLS LAST, operation ASC NULLS LAST ${page()}`; + break; + } + return request(meta, bounds, `${cte}\n${select}`); +} +function request(meta: ErrorQueryMeta, bounds: ErrorBounds, sql: string): QueryRequest { + // Machine-readable contract for the explicitly selected demo adapter, never a live endpoint. + const marker = encodeURIComponent(JSON.stringify(meta)); + return { + ...apmTransportBounds(bounds), + sql: `/* ${ERROR_QUERY_MARKER}:${marker} */\n${sql}`, + }; +} + +/** Raw stored records (SELECT *) of one log issue for the SQL explorer, so + * stack traces and other fields stay inspectable. Uses the same time, severity + * and coalesced key expressions as the aggregates; run it with the transport + * bounds from apmTransportBounds(bounds). Log datasets only. */ +export function buildErrorExplorerSql( + options: ErrorTarget & { issue: ErrorIssueKey }, + limit: number = ERROR_EXPLORE_LIMIT.default, +): string { + return buildErrorQuery({ ...options, kind: 'explore', limit }).sql; +} + +function fail(key: string): never { + throw new Error(`Malformed error analytics result: ${key}`); +} +function count(row: LogRecord, key: string): number { + const value = row[key]; + if ((typeof value !== 'number' && typeof value !== 'string') || value === '') fail(key); + const n = Number(value); + if (!Number.isSafeInteger(n) || n < 0) fail(key); + return n; +} +function nullableText(row: LogRecord, key: string): string | null { + const value = row[key]; + if (value === null || value === undefined) return null; + if (typeof value !== 'string' || value === '') fail(key); + return value; +} +function time(row: LogRecord, key: string): string { + const value = row[key]; + if (typeof value !== 'string' || !Number.isFinite(parseEventTimestamp(value))) fail(key); + return value; +} +function rows(result: unknown): LogRecord[] { + if ( + !Array.isArray(result) || + result.some((row) => row === null || typeof row !== 'object' || Array.isArray(row)) + ) + fail('rows'); + return result as LogRecord[]; +} + +export function decodeErrorSummary(result: unknown): ErrorSummary { + const list = rows(result); + if (list.length !== 1) fail('summary'); + const row = list[0]; + const occurrences = count(row, 'occurrences'), + issues = count(row, 'issues'); + if (issues > occurrences || (occurrences > 0 && issues === 0)) fail('issues'); + if (occurrences === 0) { + if (row.firstSeen != null || row.lastSeen != null) fail('firstSeen'); + return { occurrences, issues, firstSeen: null, lastSeen: null }; + } + const firstSeen = time(row, 'firstSeen'), + lastSeen = time(row, 'lastSeen'); + if (parseEventTimestamp(firstSeen) > parseEventTimestamp(lastSeen)) fail('lastSeen'); + return { occurrences, issues, firstSeen, lastSeen }; +} +/** Dense buckets covering the range; absent buckets are zero within a successful query. */ +export function decodeErrorSeries(result: unknown, bounds: ErrorBounds): ErrorBucket[] { + const normalized = timeBounds(bounds); + const start = Date.parse(normalized.startTime), + end = Date.parse(normalized.endTime), + interval = errorBucketInterval(normalized); + const buckets = Array.from({ length: Math.ceil((end - start) / interval) }, (_, index) => ({ + bucket: new Date(start + index * interval).toISOString(), + occurrences: 0, + })); + const seen = new Set(); + for (const row of rows(result)) { + const at = parseEventTimestamp(time(row, 'bucket')); + const index = (at - start) / interval; + if (!Number.isInteger(index) || index < 0 || index >= buckets.length || seen.has(index)) + fail('bucket'); + seen.add(index); + const occurrences = count(row, 'occurrences'); + if (occurrences === 0) fail('occurrences'); + buckets[index].occurrences = occurrences; + } + return buckets; +} +export function decodeErrorIssue(row: LogRecord): ErrorIssue { + const key = { + service: nullableText(row, 'service'), + errorType: nullableText(row, 'errorType'), + message: nullableText(row, 'message'), + }; + const occurrences = count(row, 'occurrences'); + if (occurrences < 1) fail('occurrences'); + const firstSeen = time(row, 'firstSeen'), + lastSeen = time(row, 'lastSeen'); + if (parseEventTimestamp(firstSeen) > parseEventTimestamp(lastSeen)) fail('lastSeen'); + return { id: errorIssueId(key), ...key, occurrences, firstSeen, lastSeen }; +} +export function decodeErrorOccurrence(row: LogRecord): ErrorOccurrence { + return { + time: time(row, 'time'), + traceId: nullableText(row, 'traceId'), + spanId: nullableText(row, 'spanId'), + operation: nullableText(row, 'operation'), + }; +} +function paged(list: LogRecord[], decode: (row: LogRecord) => T, offset: number, limit: number) { + if (list.length > limit + 1) fail('page'); + return { + rows: list.slice(0, limit).map(decode), + offset, + hasMore: list.length > limit, + } satisfies ErrorPage; +} + +type Client = Pick; +export async function loadErrorSummary(client: Client, target: ErrorTarget, signal?: AbortSignal) { + return decodeErrorSummary( + await client.query(buildErrorQuery({ ...target, kind: 'summary' }), signal), + ); +} +export async function loadErrorSeries( + client: Client, + target: ErrorTarget & { issue?: ErrorIssueKey }, + signal?: AbortSignal, +) { + return decodeErrorSeries( + await client.query(buildErrorQuery({ ...target, kind: 'series' }), signal), + resolveBounds(target), + ); +} +export async function loadErrorIssues( + client: Client, + target: ErrorTarget & { sort?: ErrorSort; offset?: number; limit?: number }, + signal?: AbortSignal, +): Promise> { + const request = buildErrorQuery({ ...target, kind: 'issues' }); + const result = rows(await client.query(request, signal)); + const page = paged(result, decodeErrorIssue, target.offset ?? 0, target.limit ?? ERROR_PAGE_SIZE); + if (new Set(page.rows.map((issue) => issue.id)).size !== page.rows.length) fail('issue identity'); + return page; +} +export async function loadErrorOccurrences( + client: Client, + target: ErrorTarget & { issue: ErrorIssueKey; offset?: number; limit?: number }, + signal?: AbortSignal, +): Promise> { + const request = buildErrorQuery({ ...target, kind: 'occurrences' }); + return paged( + rows(await client.query(request, signal)), + decodeErrorOccurrence, + target.offset ?? 0, + target.limit ?? ERROR_PAGE_SIZE, + ); +} +export async function loadErrorOverview( + client: Client, + target: ErrorTarget & { sort?: ErrorSort; offset?: number }, + signal?: AbortSignal, +) { + const { sort, offset, ...base } = target; + const [summary, series, issues] = await Promise.all([ + loadErrorSummary(client, base, signal), + loadErrorSeries(client, base, signal), + loadErrorIssues(client, { ...base, sort, offset }, signal), + ]); + return { summary, series, issues }; +} +export type ErrorOverview = Awaited>; diff --git a/frontend/src/lib/errorAnalyticsDemo.ts b/frontend/src/lib/errorAnalyticsDemo.ts new file mode 100644 index 000000000..d8a5a1f81 --- /dev/null +++ b/frontend/src/lib/errorAnalyticsDemo.ts @@ -0,0 +1,318 @@ +import { parseEventTimestamp } from '../components/explorer/timestamp'; +import { + buildErrorQuery, + errorBucketInterval, + ERROR_LEVEL_NUMBER, + ERROR_LEVEL_NUMBER_FIELD, + ERROR_QUERY_MARKER, + ERROR_SEVERITY_NUMBER, + ERROR_SEVERITY_TEXT, + type ErrorIssueKey, + type ErrorQueryMeta, + type ErrorSchema, +} from './errorAnalytics'; +import type { LogRecord, QueryRequest } from './types'; + +type Occurrence = { + service: string | null; + error_type: string | null; + message: string | null; + at: number; + trace_id: string | null; + span_id: string | null; + operation: string | null; +}; + +const MARKER = new RegExp(`^/\\* ${ERROR_QUERY_MARKER}:([^\\n]+) \\*/`); + +/** The dataset and every column the canonical query references. */ +export function errorDemoQueryTarget(request: QueryRequest) { + const match = MARKER.exec(request.sql); + if (!match) return undefined; + const meta = JSON.parse(decodeURIComponent(match[1])) as ErrorQueryMeta; + const { schema } = meta; + const fields = [ + schema.timeColumn, + ...schema.service, + ...schema.errorType, + ...schema.message, + ...schema.exception.type, + ...schema.exception.message, + ...schema.severity.text, + ...schema.severity.number, + ...[schema.exception.gate, schema.traceId, schema.spanId, schema.operation].filter( + (field): field is string => !!field, + ), + ...(schema.source === 'traces' ? ['span_status_code'] : []), + ]; + return { dataset: meta.dataset, fields: [...new Set(fields)] }; +} + +/** CAST(x AS VARCHAR) with '' as NULL. */ +function text(value: unknown): string | null { + if (value === null || value === undefined) return null; + const result = typeof value === 'object' ? JSON.stringify(value) : String(value); + return result === '' ? null : result; +} +function coalesce(row: LogRecord, fields: string[]) { + for (const field of fields) { + const value = text(row[field]); + if (value !== null) return value; + } + return null; +} +/** SQL MAX over strings ignores NULL and compares code units. */ +function max(a: string | null, b: string | null) { + return a === null ? b : b === null || a >= b ? a : b; +} +/** ASC NULLS FIRST, code-unit order. */ +function ascNullsFirst(a: string | null, b: string | null) { + return a === b ? 0 : a === null ? -1 : b === null ? 1 : a < b ? -1 : 1; +} +function ascNullsLast(a: string | null, b: string | null) { + return a === b ? 0 : a === null ? 1 : b === null ? -1 : a < b ? -1 : 1; +} +/** TRY_CAST(x AS DOUBLE): numbers and numeric strings, never truncated. */ +function double(value: unknown) { + if (typeof value === 'number') return value; + // DataFusion trims surrounding whitespace when casting strings to DOUBLE. + const trimmed = typeof value === 'string' ? value.trim() : ''; + return /^[+-]?(?:\d+\.?\d*|\.\d+)(?:[eE][+-]?\d+)?$/.test(trimmed) ? Number(trimmed) : NaN; +} +const severityNumbers = new Set(ERROR_SEVERITY_NUMBER); +const levelNumbers = new Set(ERROR_LEVEL_NUMBER); +const severities = new Set(ERROR_SEVERITY_TEXT); +function isLogError(row: LogRecord, schema: ErrorSchema) { + return ( + schema.severity.text.some((field) => { + const value = text(row[field]); + return ( + (value !== null && severities.has(value.trim().toUpperCase())) || + (field === ERROR_LEVEL_NUMBER_FIELD && levelNumbers.has(double(row[field]))) + ); + }) || + schema.severity.number.some((field) => { + return severityNumbers.has(double(row[field])); + }) + ); +} + +/** Same normalization as the SQL errors CTE, over the shared demo records. */ +export function demoErrorOccurrences( + rows: LogRecord[], + schema: ErrorSchema, + start: number, + end: number, +): Occurrence[] { + const inRange = (row: LogRecord) => { + const at = parseEventTimestamp(row[schema.timeColumn]); + return Number.isFinite(at) && at >= start && at < end ? at : undefined; + }; + if (schema.source === 'logs') { + const result: Occurrence[] = []; + for (const row of rows) { + const at = inRange(row); + if (at === undefined || !isLogError(row, schema)) continue; + result.push({ + service: coalesce(row, schema.service), + error_type: coalesce(row, schema.errorType), + message: coalesce(row, schema.message), + at, + trace_id: schema.traceId ? text(row[schema.traceId]) : null, + span_id: schema.spanId ? text(row[schema.spanId]) : null, + operation: null, + }); + } + return result; + } + type Candidate = { exception: boolean; error_type: string | null; message: string | null }; + type Span = Occurrence & { error: boolean; representative: Candidate }; + /** ROW_NUMBER() = 1 ordered by is_exception DESC, error_type, message ASC NULLS LAST. */ + const better = (a: Candidate, b: Candidate) => + a.exception !== b.exception + ? a.exception + : (ascNullsLast(a.error_type, b.error_type) || ascNullsLast(a.message, b.message)) < 0; + const spans = new Map(); + const gate = schema.exception.gate; + for (const row of rows) { + const service = row['service.name'], + trace = row[schema.traceId!], + span = row[schema.spanId!]; + const at = inRange(row); + if ( + at === undefined || + typeof service !== 'string' || + !service || + typeof trace !== 'string' || + !trace || + typeof span !== 'string' || + !span + ) + continue; + const event = !gate || row[gate] === 'exception'; + const exceptionType = event ? coalesce(row, schema.exception.type) : null, + exceptionMessage = event ? coalesce(row, schema.exception.message) : null; + const exception = exceptionType !== null || exceptionMessage !== null; + const candidate: Candidate = exception + ? { exception, error_type: exceptionType, message: exceptionMessage } + : { + exception, + error_type: coalesce(row, schema.errorType), + message: coalesce(row, schema.message), + }; + const operation = schema.operation ? text(row[schema.operation]) : null; + const key = JSON.stringify([service, trace, span]); + const previous = spans.get(key); + if (!previous) { + spans.set(key, { + service, + trace_id: trace, + span_id: span, + at, + error: double(row.span_status_code) === 2, + representative: candidate, + error_type: null, + message: null, + operation, + }); + continue; + } + previous.at = Math.min(previous.at, at); + previous.error ||= double(row.span_status_code) === 2; + if (better(candidate, previous.representative)) previous.representative = candidate; + previous.operation = max(previous.operation, operation); + } + return [...spans.values()] + .filter((span) => span.error) + .map((span) => ({ + service: span.service, + error_type: span.representative.error_type, + message: span.representative.message, + at: span.at, + trace_id: span.trace_id, + span_id: span.span_id, + operation: span.operation, + })); +} + +const matches = (item: Occurrence, issue: ErrorIssueKey) => + item.service === issue.service && + item.error_type === issue.errorType && + item.message === issue.message; +const iso = (ms: number) => new Date(ms).toISOString(); + +/** Called only by the demo client's query dispatcher. Returns undefined for + * another query family; rejects anything but the canonical generated SQL. */ +export function handleErrorDemoQuery( + request: QueryRequest, + rows: LogRecord[], +): LogRecord[] | undefined { + const match = MARKER.exec(request.sql); + if (!match) return undefined; + const meta = JSON.parse(decodeURIComponent(match[1])) as ErrorQueryMeta; + const expected = buildErrorQuery(meta); + if ( + expected.sql !== request.sql || + expected.startTime !== request.startTime || + expected.endTime !== request.endTime + ) + throw new Error('Unsupported demo error query'); + const start = Date.parse(meta.startTime), + end = Date.parse(meta.endTime); + if (meta.kind === 'explore') { + const { schema, issue } = meta; + const time = (row: LogRecord) => parseEventTimestamp(row[schema.timeColumn]); + return rows + .filter((row) => { + const at = time(row); + return ( + Number.isFinite(at) && + at >= start && + at < end && + isLogError(row, schema) && + coalesce(row, schema.service) === issue!.service && + coalesce(row, schema.errorType) === issue!.errorType && + coalesce(row, schema.message) === issue!.message + ); + }) + .sort((a, b) => time(b) - time(a)) + .slice(0, meta.limit) + .map((row) => structuredClone(row)); + } + const all = demoErrorOccurrences(rows, meta.schema, start, end); + const items = meta.issue ? all.filter((item) => matches(item, meta.issue!)) : all; + const groups = new Map(); + for (const item of all) { + const id = JSON.stringify([item.service, item.error_type, item.message]); + const group = groups.get(id) ?? { key: item, items: [] }; + group.items.push(item); + groups.set(id, group); + } + const issues = [...groups.values()].map(({ key, items }) => ({ + service: key.service, + errorType: key.error_type, + message: key.message, + occurrences: items.length, + first: Math.min(...items.map((item) => item.at)), + last: Math.max(...items.map((item) => item.at)), + })); + const page = (list: T[]) => list.slice(meta.offset!, meta.offset! + meta.limit! + 1); + switch (meta.kind) { + case 'summary': + return [ + { + occurrences: all.length, + issues: issues.length, + firstSeen: issues.length ? iso(Math.min(...issues.map((issue) => issue.first))) : null, + lastSeen: issues.length ? iso(Math.max(...issues.map((issue) => issue.last))) : null, + }, + ]; + case 'series': { + const interval = errorBucketInterval(meta); + const counts = new Map(); + for (const item of items) { + const bucket = start + Math.floor((item.at - start) / interval) * interval; + counts.set(bucket, (counts.get(bucket) ?? 0) + 1); + } + return [...counts] + .sort(([a], [b]) => a - b) + .map(([bucket, occurrences]) => ({ bucket: iso(bucket), occurrences })); + } + case 'issues': { + type Issue = (typeof issues)[number]; + const primary: Record number> = { + frequency: (a, b) => b.occurrences - a.occurrences || b.last - a.last, + latest: (a, b) => b.last - a.last || b.occurrences - a.occurrences, + first: (a, b) => b.first - a.first || b.occurrences - a.occurrences, + }; + return page( + issues.sort( + (a, b) => + primary[meta.sort!](a, b) || + ascNullsFirst(a.service, b.service) || + ascNullsFirst(a.errorType, b.errorType) || + ascNullsFirst(a.message, b.message), + ), + ).map(({ first, last, ...issue }) => ({ + ...issue, + firstSeen: iso(first), + lastSeen: iso(last), + })); + } + case 'occurrences': + return page( + items.sort( + (a, b) => + b.at - a.at || + ascNullsLast(a.trace_id, b.trace_id) || + ascNullsLast(a.span_id, b.span_id) || + ascNullsLast(a.operation, b.operation), + ), + ).map((item) => ({ + time: iso(item.at), + traceId: item.trace_id, + spanId: item.span_id, + operation: item.operation, + })); + } +} diff --git a/frontend/story-tests/native-screens.spec.ts b/frontend/story-tests/native-screens.spec.ts index bdb8f254c..623768af8 100644 --- a/frontend/story-tests/native-screens.spec.ts +++ b/frontend/story-tests/native-screens.spec.ts @@ -19,6 +19,33 @@ async function accessible(page: Page) { ).toEqual([]); } +for (const [id, source, expectedRows] of [ + ['operations-errors--trace-issue-detail', 'trace', 8], + ['operations-errors--log-issue-detail', 'log', 25], +] as const) { + test(`${id} exposes accessible occurrence details and restores focus on close`, async ({ + page, + }) => { + await story(page, id); + const dialog = page.getByRole('dialog'); + await expect(dialog).toBeVisible(); + await expect(dialog.getByTestId('error-occurrence-table').locator('tbody tr')).toHaveCount( + expectedRows, + ); + if (source === 'trace') { + await expect(dialog.getByRole('link', { name: /Open trace/ })).toHaveCount(expectedRows); + } else { + await expect(dialog.getByRole('link', { name: /SQL editor/ })).toBeVisible(); + await dialog.getByRole('button', { name: 'Next page', exact: true }).click(); + await expect(dialog.getByTestId('error-occurrence-table').locator('tbody tr')).toHaveCount(6); + } + await accessible(page); + await page.keyboard.press('Escape'); + await expect(dialog).toHaveCount(0); + await expect(page.locator('.errors-issue-button:focus')).toHaveCount(1); + }); +} + const states = [ ['agents-telemetry--default', 'Agents'], ['agents-telemetry--empty', 'No agent telemetry in this range'], @@ -47,7 +74,10 @@ const states = [ ['operations-errors--loading', 'Errors'], ['operations-errors--error', 'Could not load data'], ['operations-errors--permission', 'Permission denied'], - ['operations-errors--empty', 'No error spans'], + ['operations-errors--empty', 'No errors'], + ['operations-errors--logs', 'Errors'], + ['operations-errors--unsupported-schema', 'No eligible error dataset'], + ['operations-errors--metadata-unavailable', 'Could not verify dataset'], ['onboarding-setup-guide--empty', '1. Connect your data'], ['onboarding-setup-guide--available-datasets', '1. Connect your data'], ['onboarding-setup-guide--metadata-unavailable', '1. Connect your data'], @@ -58,11 +88,14 @@ const states = [ for (const [id, heading] of states) { test(`${id} renders its declared state and passes accessibility`, async ({ page }) => { await story(page, id); - await expect(page.getByRole('heading', { name: heading, exact: true })).toBeVisible(); + const title = page.getByRole('heading', { name: heading, exact: true }); + // Errors has independent summary, frequency and issue requests. Several + // panels can correctly expose the same failure at the same time. + await expect(id.startsWith('operations-errors--') ? title.first() : title).toBeVisible(); if (id.endsWith('--loading') && !id.startsWith('settings-network-access')) - await expect(page.getByRole('status').filter({ hasText: 'Loading data' })).toContainText( - 'Loading data', - ); + await expect( + page.getByRole('status').filter({ hasText: 'Loading data' }).first(), + ).toContainText('Loading data'); if ( id === 'agents-telemetry--permission-denied' || id === 'agents-telemetry--error' || @@ -210,6 +243,7 @@ test('error span sorting and range changes issue the matching transport requests page, }) => { await story(page, 'operations-errors--success'); + await page.getByRole('tab', { name: 'Error spans', exact: true }).click(); await expect(page.getByRole('table')).toBeVisible(); const requests = async () => JSON.parse((await page.getByTestId('error-list-requests').textContent()) ?? '[]') as Array<{