From f05d4bd468df552bea3c4af3837fee5ae2428033 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Thu, 16 Jul 2026 17:05:40 +0300 Subject: [PATCH 01/10] Add design proposal: Distributed tracing via OTLP and VictoriaTraces Signed-off-by: Alexey Artamonov --- .../distributed-tracing/README.md | 211 ++++++++++++++++++ 1 file changed, 211 insertions(+) create mode 100644 design-proposals/distributed-tracing/README.md diff --git a/design-proposals/distributed-tracing/README.md b/design-proposals/distributed-tracing/README.md new file mode 100644 index 0000000..f86c260 --- /dev/null +++ b/design-proposals/distributed-tracing/README.md @@ -0,0 +1,211 @@ + +# Distributed tracing in the Cozystack monitoring stack + +- **Title:** `Distributed tracing for managed applications via OTLP and VictoriaTraces` +- **Author(s):** `@scooby87` +- **Date:** `2026-07-16` +- **Status:** Draft + +## Overview + +Cozystack ships two of the three observability signals out of the box — metrics (VictoriaMetrics) and logs (VictoriaLogs) — but has no supported way to collect distributed **traces**. An operator who wants to see how a request flows through a managed database or messaging cluster, or to correlate a slow span with the logs and metrics it produced, has nothing to turn on. This proposal adds the third signal: an OTLP ingest path, a VictoriaTraces backend that mirrors the existing VictoriaLogs deployment one-for-one, a Grafana traces datasource wired for trace↔logs↔metrics correlation, and a per-application opt-in toggle so a tenant enables tracing on exactly the workloads that need it. + +The design is deliberately conservative: it reuses the multi-tenant topology, the operator-driven provisioning, the Grafana datasource pattern, and the values surface that metrics and logs already established, so tracing lands as "the same thing again for a third signal" rather than a new subsystem with new conventions. The backend choice — VictoriaTraces — falls out of that principle: the `VTCluster`/`VTSingle` CRDs already ship in the victoria-metrics-operator that Cozystack deploys today, so no new operator is introduced. + +## Scope and related proposals + +In scope: a traces backend (platform-wide and per-tenant), an OTLP ingest gateway, a Grafana datasource with correlation, and a per-app opt-in surface. Out of scope: automatic instrumentation of arbitrary tenant workloads, and tracing the internals of Virtual Machines or the Kubernetes control plane (VMs and Kubernetes are not traced by this proposal — only Cozystack-managed applications that can emit OTLP are). + +- **Sibling stack:** the platform monitoring stack `packages/system/monitoring` and the per-tenant stack `packages/extra/monitoring` (wired by `packages/apps/tenant/templates/monitoring.yaml`). This proposal extends both. +- **Collection agents:** `packages/system/monitoring-agents` (fluent-bit, vmagent) — the deployment pattern the OTLP collector follows. +- **Prior art in-repo:** Harbor already exposes an internal trace config (`packages/system/harbor/charts/harbor/values.yaml`, provider `jaeger` or `otel`). It is app-local and not a platform backend; this proposal supersedes ad-hoc per-app trace endpoints with a shared destination. +- **Driver:** requested by a client (hidora) who needs request-level visibility across managed DBaaS and messaging services. + +## Context + +Cozystack's observability is multi-tenant with a central backend. The platform stack runs in `tenant-root` (namespace `cozy-monitoring`) and hosts VictoriaMetrics, VictoriaLogs, Grafana (via grafana-operator), Alerta and vmalert. Tenants either ship signals to the central backend (ExternalName redirects in tenant namespaces point at the root stack; vmagent stamps a `tenant:` external label) or run their own isolated stack from `packages/extra/monitoring`. + +Metrics storage is declared as a list of tiers in values and rendered into VictoriaMetrics CRs — `metricsStorages` in `packages/system/monitoring/values.yaml` defines a `shortterm` (3d) and a `longterm` (14d) tier. Logs storage follows the identical shape: `logsStorages` renders one `VLCluster` per entry in `packages/system/monitoring/templates/vlogs/vlogs.yaml`, with the retention period set on `vlstorage.retentionPeriod`, `managedMetadata` labels for application ownership, a label stamped on the storage PVC claim template so the post-delete cleanup hook can find it, and a load-bearing guard that **fails the render** when the list is empty rather than silently shipping to a non-existent endpoint (the fix for issue `cozystack/cozystack#3181`). + +Grafana datasources are provisioned as `GrafanaDatasource` CRs by grafana-operator, one per storage: `packages/system/monitoring/templates/vm/grafana-datasource.yaml` (type `prometheus`, per metrics tier) and `packages/system/monitoring/templates/vlogs/grafana-datasource.yaml` (type `victoriametrics-logs-datasource`, per logs storage). Every datasource attaches to Grafana through `instanceSelector: { matchLabels: { dashboards: grafana } }`. + +Applications expose metrics today but there is no tracing surface. Most managed engines are scraped unconditionally through a `WorkloadMonitor` CR (clickhouse, kafka, rabbitmq, nats, mariadb) or a native operator mechanism (postgres/CNPG `enablePodMonitor`, redis via a `redis_exporter` sidecar + `VMServiceScrape`). The one app with an explicit observability toggle is foundationdb: `monitoring.enabled` in `packages/apps/foundationdb/values.yaml` gates whether its `WorkloadMonitor` renders — that toggle is the shape a `tracing.enabled` switch should copy. + +Crucially, the tracing backend needs no new operator. The victoria-metrics-operator Cozystack already runs (appVersion `v0.68.4`, `packages/system/victoria-metrics-operator`) ships the VictoriaTraces CRDs `VTCluster` and `VTSingle` (`packages/system/victoria-metrics-operator/charts/victoria-metrics-operator/crd.yaml`, CRDs `vtclusters.operator.victoriametrics.com` and `vtsingles.operator.victoriametrics.com`). `VTCluster` decomposes into `VTInsert`/`VTStorage`/`VTSelect` — a direct analog of `VLCluster`'s `vlinsert`/`vlstorage`/`vlselect` — so the render template can be lifted from `vlogs.yaml` almost verbatim. + +### The problem + +- "A query against my managed Postgres is slow and I can't see where the time goes — which statement, which replica, which downstream call." There is no trace to open. +- "I have a log line for a failed request and a latency spike on a dashboard, but no way to jump from either to the actual request span." The signals don't correlate. +- "My application already emits OTLP spans, but Cozystack gives me nowhere to send them." There is no OTLP endpoint and no backend. + +## Goals + +- Accept traces over **OTLP** (gRPC `4317` and HTTP `4318`), the standard cloud-native tracing protocol. +- Store traces in a **VictoriaTraces** backend with a **configurable retention period, defaulting to 14 days**. +- Provide a **per-application opt-in** `tracing.enabled` toggle; tracing is **off by default** and adds zero overhead until enabled. +- Provision a **Grafana traces datasource** and wire **trace↔logs↔metrics** correlation so an operator can pivot between all three signals. +- Preserve **per-tenant isolation and the central-backend topology** exactly as metrics and logs do today. +- Introduce **no new operator** and no new provisioning convention — reuse VictoriaTraces (already in vm-operator), the storage-list values shape, and the grafana-operator datasource pattern. + +### Non-goals + +- Auto-instrumenting arbitrary tenant workloads. This proposal wires the *transport and storage*; emitting spans is the application's job (native where the engine supports it, sidecar/agent otherwise). +- Tracing Virtual Machines or Kubernetes control-plane internals. +- Changing the existing metrics or logs pipelines. +- Mandating a sampling policy for tenant applications (the platform sets a safe default and exposes a knob). + +## Design + +### 1. Backend: VictoriaTraces (`tracingStorages`) + +Add a `tracingStorages` list to the monitoring values, parallel to `metricsStorages` and `logsStorages`, in both `packages/system/monitoring/values.yaml` and `packages/extra/monitoring/values.yaml`: + +```yaml +tracingStorages: +- name: generic + retentionPeriod: "14d" # configurable; default 14 days per the requirement + storage: 10Gi + storageClassName: "" +``` + +Render one `VTCluster` per entry in a new `templates/vtraces/vtraces.yaml`, lifted from `templates/vlogs/vlogs.yaml`: `managedMetadata` labels for application ownership, replica counts on `vtinsert`/`vtselect`/`vtstorage`, `vtstorage.retentionPeriod` from the entry, the storage-PVC claim-template label for the release-scoped post-delete cleanup hook, and — reusing the `cozystack/cozystack#3181` lesson — a guard that **fails the render on an empty list** so a misconfiguration is loud, not a silent black hole for spans. + +```yaml +{{- range .Values.tracingStorages }} +--- +apiVersion: operator.victoriametrics.com/v1 +kind: VTCluster +metadata: + name: {{ .name }} +spec: + managedMetadata: + labels: + apps.cozystack.io/application.kind: Monitoring + apps.cozystack.io/application.name: {{ $.Release.Name }} + vtinsert: { replicaCount: 2 } + vtselect: { replicaCount: 2 } + vtstorage: + retentionPeriod: {{ .retentionPeriod | quote }} + replicaCount: 2 + storage: + volumeClaimTemplate: + metadata: + labels: + apps.cozystack.io/application.name: {{ $.Release.Name }} + spec: + {{- with .storageClassName }} + storageClassName: {{ . }} + {{- end }} + resources: + requests: + storage: {{ .storage }} +{{- end }} +``` + +The monitoring HelmRelease readiness gate keys on the new `VTCluster` the same way it keys on `VLCluster` today. + +### 2. OTLP ingest via an OpenTelemetry Collector + +VictoriaTraces ingests OTLP natively, but a collector in front gives the platform a stable tenant-facing endpoint, sampling/rate-limiting, and a place to attach the `tenant:` resource attribute — the same role vmagent and fluent-bit play for metrics and logs. Deploy an **OpenTelemetry Collector** in `cozy-monitoring`, following the `packages/system/monitoring-agents` deployment/service pattern: + +- Receivers: `otlp` on gRPC `4317` and HTTP `4318`. +- Processors: `batch`, a `tail_sampling`/`probabilistic_sampler` governed by the platform default, and `resource` to stamp `tenant`. +- Exporter: OTLP to the `vtinsert` service of the `tracingStorages` backend. + +Tenants target an in-cluster OTLP endpoint (e.g. `otel-collector.cozy-monitoring.svc`); per-tenant namespaces get an ExternalName redirect to the root collector, mirroring the existing `vlinsert-generic` logs redirect, so the central-vs-isolated choice works identically for traces. + +### 3. Grafana datasource and correlation + +Add a `GrafanaDatasource` CR per `tracingStorages` entry in `templates/vtraces/grafana-datasource.yaml`, mirroring the logs datasource template and attaching through `instanceSelector: { matchLabels: { dashboards: grafana } }`. VictoriaTraces exposes a Jaeger-compatible query API, so the datasource is `type: jaeger` (or the dedicated VictoriaTraces datasource plugin, allow-listed like `victoriametrics-logs-datasource` is today) pointed at the `vtselect` service. Configure: + +- **Trace → logs**: link to the VictoriaLogs datasource keyed on `trace_id`. +- **Trace → metrics**: link to the VictoriaMetrics datasource for RED-style span metrics. +- **Logs/metrics → trace**: derived fields on the existing datasources so a `trace_id` in a log or exemplar opens the trace. + +### 4. Per-application opt-in + +Add a `tracing` struct to each participating app's `values.yaml` using the cozyvalues-gen annotation conventions, modelled on foundationdb's `monitoring.enabled` toggle and postgres's `backup` struct: + +```yaml +## @typedef {struct} Tracing - OpenTelemetry (OTLP) tracing configuration. +## @field {bool} enabled - Enable OTLP trace export from this application. +## @field {string} [endpoint] - OTLP collector endpoint. Defaults to the platform collector in cozy-monitoring. +## @field {string} [samplingRatio] - Head-sampling ratio 0.0..1.0. Defaults to the platform policy. + +## @param {Tracing} tracing - OpenTelemetry tracing configuration. +tracing: + enabled: false + endpoint: "" + samplingRatio: "" +``` + +How an app emits spans depends on the engine: + +- **Native OTLP**, wired by chart config: ClickHouse (`opentelemetry_span_log`, currently disabled in `packages/apps/clickhouse/templates/clickhouse.yaml`) and NATS (native OTLP in recent versions). +- **Sidecar/agent OTLP**: Kafka and RabbitMQ (JVM/plugin agents), MariaDB, Redis and Postgres (an OpenTelemetry agent/exporter sidecar). The `tracing.enabled` toggle gates the sidecar and the `OTEL_EXPORTER_OTLP_ENDPOINT` env, defaulting to the platform collector. + +### 5. Data flow + +```mermaid +flowchart LR + app["Managed app
(tracing.enabled)"] -- OTLP 4317/4318 --> col["OpenTelemetry Collector
cozy-monitoring"] + col -- OTLP --> vt["VictoriaTraces
VTCluster (vtinsert→vtstorage)"] + gr["Grafana"] -- Jaeger query --> vt + gr -. trace_id .-> vl["VictoriaLogs"] + gr -. span metrics .-> vm["VictoriaMetrics"] +``` + +## User-facing changes + +- A new `tracingStorages` block in the monitoring values (system and per-tenant), with `retentionPeriod` defaulting to 14 days. +- A new per-app `tracing.*` block; off by default. +- A Traces datasource and Explore/Traces view in Grafana, with pivot links to logs and metrics. +- A docs entry point (`docs/observability/distributed-tracing.md` in `cozystack/cozystack`) covering how to enable tracing and point an app at the collector. + +## Upgrade and rollback compatibility + +The change is purely additive and opt-in. Existing clusters see no behavioural change until `tracingStorages` is set and an app flips `tracing.enabled`. An empty/absent `tracingStorages` renders no `VTCluster` (guarded, so it fails loudly only if a downstream component is told to expect one — matching the logs behaviour). No data migration is required. Rollback is removing the `tracingStorages` block and the per-app toggles; trace data in VictoriaTraces PVCs is discarded on backend removal (flagged: irreversible for already-stored spans, like logs). + +## Security + +- **New tenant-supplied input**: the OTLP endpoint accepts spans from tenant workloads. The collector is the trust boundary — it enforces per-tenant `tenant` resource attribution, rate-limits, and sampling to prevent a noisy or hostile tenant from exhausting the shared backend. +- **Isolation**: central-backend mode keeps the existing tenant-labelling model; isolated mode (`packages/extra/monitoring`) keeps traces inside the tenant. +- **Transport**: OTLP endpoints should be TLS-terminated; align with the unified TLS/PKI model (`design-proposals/unified-tls-pki`) rather than minting bespoke certs. +- **RBAC**: new `VTCluster`/`GrafanaDatasource`/collector resources need the same narrowly-scoped RBAC the metrics/logs equivalents already have. No new secret classes are introduced beyond the OTLP endpoint credentials, if any. + +## Failure and edge cases + +- Empty `tracingStorages` while a consumer expects a backend → loud render failure (mirrors `cozystack/cozystack#3181`), never a silent span black hole. +- Collector unreachable from an app → the app's exporter drops/retries per OTLP defaults; the app itself does not crash and serving is unaffected. +- Backend storage exhausted → oldest traces evicted per `retentionPeriod`; ingest backpressures at the collector, not the app. +- `tracing.enabled: false` → no sidecar, no env, no CR: zero overhead. +- App emits OTLP but no backend deployed → collector accepts and drops (or the toggle is guarded to require a backend); documented, not surprising. + +## Testing + +- **Unit/lint**: `helm template` + `helm lint` for the new `vtraces` templates and each app's `tracing` block; assert `VTCluster`, the collector, and the `GrafanaDatasource` render only when configured, and that an empty `tracingStorages` fails the render. +- **e2e** (Chainsaw, per `docs/agents/e2e-testing.md`): deploy the monitoring stack with a `tracingStorages` entry, deploy one app (start with a native-OTLP engine, e.g. ClickHouse) with `tracing.enabled: true`, generate activity, then assert a trace is queryable via the `vtselect` Jaeger API and visible in Grafana. +- **Manual**: verify trace→logs and trace→metrics pivots in Grafana. + +## Rollout + +1. **Backend + ingest**: `tracingStorages`/`VTCluster` and the OpenTelemetry Collector in `packages/system/monitoring` and `packages/extra/monitoring`. No app changes yet. +2. **Grafana**: traces datasource + correlation links. +3. **Per-app toggles**: start with native-OTLP engines (ClickHouse, NATS), then sidecar-based engines (Kafka, RabbitMQ, MariaDB, Redis, Postgres), one PR per app. +4. **Docs**: enablement guide under `docs/observability/`. + +## Open questions + +- **VictoriaTraces maturity**: the CRDs ship in the operator, but is VictoriaTraces production-ready at the version Cozystack pins? If not, Grafana Tempo is the drop-in fallback (see Alternatives) — the collector and per-app surfaces are backend-agnostic, so only the backend template and datasource type change. +- **`VTCluster` vs `VTSingle`** as the default: cluster for HA parity with metrics/logs, or single for a lighter footprint on small clusters? +- **Sampling**: head sampling at the app vs tail sampling at the collector; what platform default? +- **Collector deployment**: Deployment (gateway) vs DaemonSet (agent) — gateway matches the central-backend model; DaemonSet matches fluent-bit. +- **External OTLP exposure**: should tenants be able to push spans from outside the cluster, and if so through which ingress/Gateway path? + +## Alternatives considered + +- **Grafana Tempo** (backend): mature, object-storage-backed (cheap retention on the seaweedfs/COSI storage Cozystack already runs), and the strongest Grafana-native correlation story. Rejected as the *primary* choice only to keep the stack single-vendor (VictoriaMetrics/Logs/Traces share one operator and one operational model). It remains the recommended fallback if VictoriaTraces proves immature — the rest of this design is unchanged by the swap. +- **Jaeger** (backend): mature and OTLP-native, but its own UI and weaker Grafana integration cut against the single-pane correlation goal, and it adds an operator/storage story Cozystack doesn't already have. +- **No collector, app → backend directly** (ingest): simpler, but loses the shared trust boundary, per-tenant attribution, and central sampling/rate-limiting; rejected for the same reasons metrics go through vmagent and logs through fluent-bit rather than writing to storage directly. +- **Always-on tracing** (opt-in model): rejected — tracing overhead and storage cost must be a tenant's explicit choice; default off matches the requirement and the principle of least surprise. From bd75a12b80fc0289c513a99804e39ef4c9db3338 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Thu, 16 Jul 2026 17:25:42 +0300 Subject: [PATCH 02/10] =?UTF-8?q?Refine=20tracing=20design:=20staged=20A?= =?UTF-8?q?=E2=86=92B=20ingest=20grounded=20in=20current=20architecture?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Direct-to-vtinsert MVP (mirrors logs) then an opt-in OpenTelemetry Collector gateway; ExternalName redirect via cozystack-basics; poller readiness gate on VTCluster; clarify WorkloadMonitor is operational-only so tracing opt-in lives in app values. Signed-off-by: Alexey Artamonov --- .../distributed-tracing/README.md | 42 ++++++++++++------- 1 file changed, 26 insertions(+), 16 deletions(-) diff --git a/design-proposals/distributed-tracing/README.md b/design-proposals/distributed-tracing/README.md index f86c260..c511e69 100644 --- a/design-proposals/distributed-tracing/README.md +++ b/design-proposals/distributed-tracing/README.md @@ -29,7 +29,7 @@ Metrics storage is declared as a list of tiers in values and rendered into Victo Grafana datasources are provisioned as `GrafanaDatasource` CRs by grafana-operator, one per storage: `packages/system/monitoring/templates/vm/grafana-datasource.yaml` (type `prometheus`, per metrics tier) and `packages/system/monitoring/templates/vlogs/grafana-datasource.yaml` (type `victoriametrics-logs-datasource`, per logs storage). Every datasource attaches to Grafana through `instanceSelector: { matchLabels: { dashboards: grafana } }`. -Applications expose metrics today but there is no tracing surface. Most managed engines are scraped unconditionally through a `WorkloadMonitor` CR (clickhouse, kafka, rabbitmq, nats, mariadb) or a native operator mechanism (postgres/CNPG `enablePodMonitor`, redis via a `redis_exporter` sidecar + `VMServiceScrape`). The one app with an explicit observability toggle is foundationdb: `monitoring.enabled` in `packages/apps/foundationdb/values.yaml` gates whether its `WorkloadMonitor` renders — that toggle is the shape a `tracing.enabled` switch should copy. +Applications expose metrics today but there is no tracing surface. Most managed engines are scraped unconditionally through a `WorkloadMonitor` CR (clickhouse, kafka, rabbitmq, nats, mariadb) or a native operator mechanism (postgres/CNPG `enablePodMonitor`, redis via a `redis_exporter` sidecar + `VMServiceScrape`). The one app with an explicit observability toggle is foundationdb: `monitoring.enabled` in `packages/apps/foundationdb/values.yaml` gates whether its `WorkloadMonitor` renders — that toggle is the shape a `tracing.enabled` switch should copy. Note that `WorkloadMonitor` itself is not a fit for carrying tracing config: its controller (`internal/controller/workloadmonitor_controller.go`) reconciles it into `Workload` objects that track replicas/resources/operational status for the dashboard and billing surfaces — it does not emit scrape configs, and tracing opt-in therefore belongs in each app's `values.yaml`, not in `WorkloadMonitor`. Crucially, the tracing backend needs no new operator. The victoria-metrics-operator Cozystack already runs (appVersion `v0.68.4`, `packages/system/victoria-metrics-operator`) ships the VictoriaTraces CRDs `VTCluster` and `VTSingle` (`packages/system/victoria-metrics-operator/charts/victoria-metrics-operator/crd.yaml`, CRDs `vtclusters.operator.victoriametrics.com` and `vtsingles.operator.victoriametrics.com`). `VTCluster` decomposes into `VTInsert`/`VTStorage`/`VTSelect` — a direct analog of `VLCluster`'s `vlinsert`/`vlstorage`/`vlselect` — so the render template can be lifted from `vlogs.yaml` almost verbatim. @@ -103,17 +103,21 @@ spec: {{- end }} ``` -The monitoring HelmRelease readiness gate keys on the new `VTCluster` the same way it keys on `VLCluster` today. +The monitoring HelmRelease must gate readiness on the new `VTCluster` exactly as it does for `VLCluster` today: `waitStrategy: poller` plus a `healthCheckExprs` entry that waits for the CR's `status.updateStatus == 'operational'` (see `packages/extra/monitoring/templates/helmrelease.yaml`). Without the poller gate the release flips Ready as soon as Helm applies the CR — the exact silent-black-hole failure mode that motivated `cozystack/cozystack#3181` for logs. -### 2. OTLP ingest via an OpenTelemetry Collector +### 2. OTLP ingest (staged: direct-to-backend, then a collector gateway) -VictoriaTraces ingests OTLP natively, but a collector in front gives the platform a stable tenant-facing endpoint, sampling/rate-limiting, and a place to attach the `tenant:` resource attribute — the same role vmagent and fluent-bit play for metrics and logs. Deploy an **OpenTelemetry Collector** in `cozy-monitoring`, following the `packages/system/monitoring-agents` deployment/service pattern: +`vtinsert` accepts OTLP natively, so the ingest path mirrors logs one-for-one: where fluent-bit ships to `vlinsert-generic`, a traced application ships OTLP to `vtinsert-generic`. This proposal stages the ingest so the platform gets value immediately and grows into the collector the client asked for, with **no app-visible endpoint change between stages**. + +**Stage A (MVP) — direct to `vtinsert`.** Applications push OTLP straight to the VictoriaTraces insert service. Add an `otel-traces` (or `vtinsert-generic`) **ExternalName redirect** to `packages/system/cozystack-basics/templates/monitoring-external-services.yaml`, alongside the existing `vlinsert-generic`/`vminsert-*` entries, pointing `*.cozy-monitoring.svc` → `*.tenant-root.svc.cluster.local`, gated on the same `_cluster.monitoring-enabled` flag (from `monitoring.rootEnabled`). This is the exact mechanism logs and metrics already use; central-vs-isolated tenants work identically with zero new machinery. + +**Stage B — toggleable OpenTelemetry Collector gateway.** Add an OpenTelemetry Collector (gateway **Deployment** in `cozy-monitoring`, following the `packages/system/monitoring-agents` pattern) in front of `vtinsert` for the capabilities direct ingest can't provide: central sampling, rate-limiting, and per-`tenant` resource attribution. It is the "OpenTelemetry Collector/Agent via an option" the client explicitly requested, and it is opt-in. - Receivers: `otlp` on gRPC `4317` and HTTP `4318`. - Processors: `batch`, a `tail_sampling`/`probabilistic_sampler` governed by the platform default, and `resource` to stamp `tenant`. - Exporter: OTLP to the `vtinsert` service of the `tracingStorages` backend. -Tenants target an in-cluster OTLP endpoint (e.g. `otel-collector.cozy-monitoring.svc`); per-tenant namespaces get an ExternalName redirect to the root collector, mirroring the existing `vlinsert-generic` logs redirect, so the central-vs-isolated choice works identically for traces. +Applications always target a stable in-cluster endpoint (e.g. `…cozy-monitoring.svc:4317`); flipping from Stage A to Stage B only re-points that ExternalName from `vtinsert` to the collector, so the migration is transparent to every traced app. A gateway **Deployment** (not a DaemonSet) is correct here because OTLP is push-based over the network — apps are the agents; the collector plays the centralized `vtinsert`/vmagent role, not the node-local fluent-bit role. ### 3. Grafana datasource and correlation @@ -149,8 +153,10 @@ How an app emits spans depends on the engine: ```mermaid flowchart LR - app["Managed app
(tracing.enabled)"] -- OTLP 4317/4318 --> col["OpenTelemetry Collector
cozy-monitoring"] - col -- OTLP --> vt["VictoriaTraces
VTCluster (vtinsert→vtstorage)"] + app["Managed app
(tracing.enabled)"] -- "OTLP 4317/4318
(stable svc endpoint)" --> ext["ExternalName redirect
cozy-monitoring → tenant-root"] + ext -- "Stage A: direct" --> vt["VictoriaTraces
VTCluster (vtinsert→vtstorage)"] + ext -. "Stage B: via gateway" .-> col["OpenTelemetry Collector
(sampling / rate-limit / tenant)"] + col -- OTLP --> vt gr["Grafana"] -- Jaeger query --> vt gr -. trace_id .-> vl["VictoriaLogs"] gr -. span metrics .-> vm["VictoriaMetrics"] @@ -169,7 +175,7 @@ The change is purely additive and opt-in. Existing clusters see no behavioural c ## Security -- **New tenant-supplied input**: the OTLP endpoint accepts spans from tenant workloads. The collector is the trust boundary — it enforces per-tenant `tenant` resource attribution, rate-limits, and sampling to prevent a noisy or hostile tenant from exhausting the shared backend. +- **New tenant-supplied input**: the OTLP endpoint accepts spans from tenant workloads. In Stage A the shared `vtinsert` backend is the exposed surface (relying on VictoriaTraces' own limits and the per-tenant isolation of isolated stacks); Stage B's collector becomes the explicit trust boundary — enforcing per-tenant `tenant` resource attribution, rate-limits, and sampling so a noisy or hostile tenant cannot exhaust the shared backend. This hardening is the main reason to promote Stage B on a shared central backend. - **Isolation**: central-backend mode keeps the existing tenant-labelling model; isolated mode (`packages/extra/monitoring`) keeps traces inside the tenant. - **Transport**: OTLP endpoints should be TLS-terminated; align with the unified TLS/PKI model (`design-proposals/unified-tls-pki`) rather than minting bespoke certs. - **RBAC**: new `VTCluster`/`GrafanaDatasource`/collector resources need the same narrowly-scoped RBAC the metrics/logs equivalents already have. No new secret classes are introduced beyond the OTLP endpoint credentials, if any. @@ -177,10 +183,10 @@ The change is purely additive and opt-in. Existing clusters see no behavioural c ## Failure and edge cases - Empty `tracingStorages` while a consumer expects a backend → loud render failure (mirrors `cozystack/cozystack#3181`), never a silent span black hole. -- Collector unreachable from an app → the app's exporter drops/retries per OTLP defaults; the app itself does not crash and serving is unaffected. -- Backend storage exhausted → oldest traces evicted per `retentionPeriod`; ingest backpressures at the collector, not the app. +- OTLP endpoint unreachable from an app (backend or collector down) → the app's exporter drops/retries per OTLP defaults; the app itself does not crash and serving is unaffected. +- Backend storage exhausted → oldest traces evicted per `retentionPeriod`; ingest backpressures at the receiver (`vtinsert` in Stage A, the collector in Stage B), not the app. - `tracing.enabled: false` → no sidecar, no env, no CR: zero overhead. -- App emits OTLP but no backend deployed → collector accepts and drops (or the toggle is guarded to require a backend); documented, not surprising. +- App emits OTLP but no backend deployed → the ExternalName resolves to nothing and the exporter drops/retries harmlessly; the app toggle is documented to require a `tracingStorages` backend. ## Testing @@ -190,22 +196,26 @@ The change is purely additive and opt-in. Existing clusters see no behavioural c ## Rollout -1. **Backend + ingest**: `tracingStorages`/`VTCluster` and the OpenTelemetry Collector in `packages/system/monitoring` and `packages/extra/monitoring`. No app changes yet. +1. **Stage A — backend + direct ingest**: `tracingStorages`/`VTCluster` in `packages/system/monitoring` (+ `packages/extra/monitoring`) with the poller readiness gate, plus the `vtinsert` ExternalName redirect in `packages/system/cozystack-basics`. Apps can push OTLP directly. No collector yet. 2. **Grafana**: traces datasource + correlation links. 3. **Per-app toggles**: start with native-OTLP engines (ClickHouse, NATS), then sidecar-based engines (Kafka, RabbitMQ, MariaDB, Redis, Postgres), one PR per app. -4. **Docs**: enablement guide under `docs/observability/`. +4. **Stage B — collector gateway**: add the toggleable OpenTelemetry Collector Deployment and re-point the ExternalName from `vtinsert` to the collector (transparent to apps). Ships sampling/rate-limiting/tenant-attribution. +5. **Docs**: enablement guide under `docs/observability/`. ## Open questions - **VictoriaTraces maturity**: the CRDs ship in the operator, but is VictoriaTraces production-ready at the version Cozystack pins? If not, Grafana Tempo is the drop-in fallback (see Alternatives) — the collector and per-app surfaces are backend-agnostic, so only the backend template and datasource type change. - **`VTCluster` vs `VTSingle`** as the default: cluster for HA parity with metrics/logs, or single for a lighter footprint on small clusters? -- **Sampling**: head sampling at the app vs tail sampling at the collector; what platform default? -- **Collector deployment**: Deployment (gateway) vs DaemonSet (agent) — gateway matches the central-backend model; DaemonSet matches fluent-bit. +- **Sampling**: head sampling at the app vs tail sampling at the collector; what platform default? (Only relevant once Stage B lands.) - **External OTLP exposure**: should tenants be able to push spans from outside the cluster, and if so through which ingress/Gateway path? +- **Stage-B trigger**: what concrete signal (backend load, abuse, a tenant-attribution requirement) promotes the collector gateway from optional to default? + +Resolved during design (recorded here so they are not re-litigated): ingest is **staged A→B** — direct-to-`vtinsert` first, collector gateway as an opt-in second — because `vtinsert` is OTLP-native and the ExternalName redirect keeps the app-facing endpoint stable across the switch. The collector is a gateway **Deployment**, not a DaemonSet, because OTLP is push-based (apps are the agents). Tracing opt-in lives in app `values.yaml`, not `WorkloadMonitor` (operational-only). ## Alternatives considered - **Grafana Tempo** (backend): mature, object-storage-backed (cheap retention on the seaweedfs/COSI storage Cozystack already runs), and the strongest Grafana-native correlation story. Rejected as the *primary* choice only to keep the stack single-vendor (VictoriaMetrics/Logs/Traces share one operator and one operational model). It remains the recommended fallback if VictoriaTraces proves immature — the rest of this design is unchanged by the swap. - **Jaeger** (backend): mature and OTLP-native, but its own UI and weaker Grafana integration cut against the single-pane correlation goal, and it adds an operator/storage story Cozystack doesn't already have. -- **No collector, app → backend directly** (ingest): simpler, but loses the shared trust boundary, per-tenant attribution, and central sampling/rate-limiting; rejected for the same reasons metrics go through vmagent and logs through fluent-bit rather than writing to storage directly. +- **No collector, app → backend directly** (ingest): adopted as **Stage A**, not rejected — `vtinsert` is OTLP-native, so direct ingest is the fastest correct MVP and an exact mirror of the logs path. Its limitations (no shared trust boundary, per-tenant attribution, or central sampling/rate-limiting) are exactly what **Stage B**'s collector gateway adds later, without changing the app-facing endpoint. +- **Collector as a DaemonSet agent** (ingest topology): rejected — that node-local shape fits fluent-bit tailing log files, but OTLP traces are pushed over the network by the apps themselves, so a centralized gateway Deployment (the `vtinsert`/vmagent role) is the right shape. - **Always-on tracing** (opt-in model): rejected — tracing overhead and storage cost must be a tenant's explicit choice; default off matches the requirement and the principle of least surprise. From 39e0205a05a29b6e3d12a4e1f02dc41f6244f478 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Fri, 17 Jul 2026 10:25:38 +0300 Subject: [PATCH 03/10] Address review: tenant headers, OTLP port/path, tail-sampling affinity, disk retention - Stage A: document AccountID/ProjectID tenant-header injection and the real vtinsert OTLP contract (port 10481, /insert/opentelemetry/v1/traces); ExternalName aliases DNS only. - Stage B: otlphttp exporter sets tenant headers; tail-sampling trace affinity (single replica or loadbalancingexporter routing_key=traceID). - Backend: configurable replicaCount; retentionDiskSpaceUsage vs age retention; durability caveat (replicaCount != data replication); absent-vs-empty contract. - Grafana: span-metrics need a spanmetrics connector; read-side tenant isolation. - Security/Failure/Testing/Open questions updated accordingly. - Drop leftover authoring comment; add application.group managedMetadata label. Signed-off-by: Alexey Artamonov --- .../distributed-tracing/README.md | 68 ++++++++++++------- 1 file changed, 45 insertions(+), 23 deletions(-) diff --git a/design-proposals/distributed-tracing/README.md b/design-proposals/distributed-tracing/README.md index c511e69..e9a7138 100644 --- a/design-proposals/distributed-tracing/README.md +++ b/design-proposals/distributed-tracing/README.md @@ -1,4 +1,3 @@ - # Distributed tracing in the Cozystack monitoring stack - **Title:** `Distributed tracing for managed applications via OTLP and VictoriaTraces` @@ -64,12 +63,18 @@ Add a `tracingStorages` list to the monitoring values, parallel to `metricsStora ```yaml tracingStorages: - name: generic - retentionPeriod: "14d" # configurable; default 14 days per the requirement + retentionPeriod: "14d" # age-based retention; default 14 days per the requirement + retentionDiskSpaceUsage: "" # optional disk-based cap (e.g. "80%" / bytes); see note below storage: 10Gi storageClassName: "" + replicaCount: 2 # component scaling, NOT data replication (see durability note) ``` -Render one `VTCluster` per entry in a new `templates/vtraces/vtraces.yaml`, lifted from `templates/vlogs/vlogs.yaml`: `managedMetadata` labels for application ownership, replica counts on `vtinsert`/`vtselect`/`vtstorage`, `vtstorage.retentionPeriod` from the entry, the storage-PVC claim-template label for the release-scoped post-delete cleanup hook, and — reusing the `cozystack/cozystack#3181` lesson — a guard that **fails the render on an empty list** so a misconfiguration is loud, not a silent black hole for spans. +Render one `VTCluster` per entry in a new `templates/vtraces/vtraces.yaml`, lifted from `templates/vlogs/vlogs.yaml`: `managedMetadata` labels for application ownership, configurable replica counts on `vtinsert`/`vtselect`/`vtstorage`, `vtstorage.retentionPeriod` from the entry, the storage-PVC claim-template label for the release-scoped post-delete cleanup hook, and the `cozystack/cozystack#3181` guard (see the absent-vs-empty contract below). + +**Absent vs empty contract** (raised in review — the two must not be conflated): an **absent/unset** `tracingStorages` key means "tracing disabled", and the template renders nothing and succeeds (so tracing stays opt-in and a cluster that never wanted traces is never broken). An **explicitly empty list** (`tracingStorages: []`) is a misconfiguration — a consumer asked for tracing but declared no backend — and **fails the render**, exactly like the logs `#3181` guard. Both cases must be covered by tests. + +**Durability note** (raised in review): `replicaCount` scales the number of `vtstorage` pods for throughput/availability of the *component*, but VictoriaTraces cluster mode does **not** replicate span data between storage nodes — losing a storage node can make some queries return partial results. This is the same durability model as `VLCluster`, and it is *not* HA data replication. Real cross-node durability requires either replication through Collectors into two independent VictoriaTraces backends or an equivalent design; this is called out in Open questions rather than solved here. ```yaml {{- range .Values.tracingStorages }} @@ -81,13 +86,17 @@ metadata: spec: managedMetadata: labels: + apps.cozystack.io/application.group: apps.cozystack.io apps.cozystack.io/application.kind: Monitoring apps.cozystack.io/application.name: {{ $.Release.Name }} - vtinsert: { replicaCount: 2 } - vtselect: { replicaCount: 2 } + vtinsert: { replicaCount: {{ .replicaCount | default 2 }} } + vtselect: { replicaCount: {{ .replicaCount | default 2 }} } vtstorage: retentionPeriod: {{ .retentionPeriod | quote }} - replicaCount: 2 + {{- with .retentionDiskSpaceUsage }} + retentionMaxDiskSpaceUsagePercent: {{ . | quote }} + {{- end }} + replicaCount: {{ .replicaCount | default 2 }} storage: volumeClaimTemplate: metadata: @@ -111,22 +120,30 @@ The monitoring HelmRelease must gate readiness on the new `VTCluster` exactly as **Stage A (MVP) — direct to `vtinsert`.** Applications push OTLP straight to the VictoriaTraces insert service. Add an `otel-traces` (or `vtinsert-generic`) **ExternalName redirect** to `packages/system/cozystack-basics/templates/monitoring-external-services.yaml`, alongside the existing `vlinsert-generic`/`vminsert-*` entries, pointing `*.cozy-monitoring.svc` → `*.tenant-root.svc.cluster.local`, gated on the same `_cluster.monitoring-enabled` flag (from `monitoring.rootEnabled`). This is the exact mechanism logs and metrics already use; central-vs-isolated tenants work identically with zero new machinery. +**OTLP service contract** (raised in review — the exact ports/path matter): an `ExternalName` Service only aliases DNS; it does not remap ports or translate the OTLP/HTTP path. `vtinsert` defaults to OTLP/HTTP on port `10481` at path `/insert/opentelemetry/v1/traces`, and OTLP/gRPC only when `-otlpGRPCListenAddr` is set on the component (conventionally `4317`). So Stage A must be precise: either (a) point apps at `vtinsert`'s real OTLP/HTTP endpoint (`…:10481/insert/opentelemetry/v1/traces`) and enable the gRPC listener on the `VTCluster` if gRPC is wanted, redirecting via a `ClusterIP`/`ExternalName` Service that exposes those ports, or (b) front `vtinsert` with the Stage-B collector, which owns the canonical `4317/4318` OTLP listeners. The earlier "stable `…:4317`" shorthand assumes the collector or an explicit port-mapped Service — plain DNS aliasing alone is not enough. + +**Tenant routing needs headers** (raised in review — this is the load-bearing multi-tenancy point): VictoriaTraces attributes a tenant from the `AccountID`/`ProjectID` request headers (query-string equivalents take priority); with neither set, everything lands in the default tenant `0:0`. Neither an `ExternalName` redirect nor a collector `resource` processor sets those headers. So on a shared central backend the write path must inject `AccountID`/`ProjectID` derived from authenticated workload identity — from the app's OTLP exporter, a trusted per-tenant proxy, or (Stage B) the collector's `otlphttp` exporter `headers:` block — in addition to stamping the `tenant` resource attribute for query-time labelling. Isolated per-tenant stacks (`packages/extra/monitoring`) sidestep this by not sharing a backend. + **Stage B — toggleable OpenTelemetry Collector gateway.** Add an OpenTelemetry Collector (gateway **Deployment** in `cozy-monitoring`, following the `packages/system/monitoring-agents` pattern) in front of `vtinsert` for the capabilities direct ingest can't provide: central sampling, rate-limiting, and per-`tenant` resource attribution. It is the "OpenTelemetry Collector/Agent via an option" the client explicitly requested, and it is opt-in. -- Receivers: `otlp` on gRPC `4317` and HTTP `4318`. -- Processors: `batch`, a `tail_sampling`/`probabilistic_sampler` governed by the platform default, and `resource` to stamp `tenant`. -- Exporter: OTLP to the `vtinsert` service of the `tracingStorages` backend. +- Receivers: `otlp` on gRPC `4317` and HTTP `4318` (the collector owns these canonical ports; it translates to `vtinsert`'s `10481`/`/insert/opentelemetry/v1/traces` on export). +- Processors: `batch`, a sampler (see the affinity note), and `resource` to stamp the `tenant` attribute. +- Exporter: `otlphttp` to the `vtinsert` service of the `tracingStorages` backend, with the `headers:` block setting `AccountID`/`ProjectID` from the authenticated tenant identity (not just the resource attribute). + +**Tail-sampling affinity** (raised in review): tail sampling must see *all* spans of a trace on one instance, but a plain Kubernetes Service round-robins spans across collector replicas and would make sampling decisions on incomplete traces. So the sampling tier must run at a single replica, or use the two-tier pattern — a first tier with the `loadbalancingexporter` (`routing_key: traceID`) consistently hashing each trace to a fixed second-tier instance that runs `tail_sampling`. Head sampling (`probabilistic_sampler`) has no such constraint and can scale freely; the platform default and this trade-off are an Open question. -Applications always target a stable in-cluster endpoint (e.g. `…cozy-monitoring.svc:4317`); flipping from Stage A to Stage B only re-points that ExternalName from `vtinsert` to the collector, so the migration is transparent to every traced app. A gateway **Deployment** (not a DaemonSet) is correct here because OTLP is push-based over the network — apps are the agents; the collector plays the centralized `vtinsert`/vmagent role, not the node-local fluent-bit role. +Applications always target a stable in-cluster endpoint (e.g. `…cozy-monitoring.svc:4317` once the collector fronts ingest); flipping from Stage A to Stage B only re-points that ExternalName from `vtinsert` to the collector, so the migration is transparent to every traced app. A gateway **Deployment** (not a DaemonSet) is correct here because OTLP is push-based over the network — apps are the agents; the collector plays the centralized `vtinsert`/vmagent role, not the node-local fluent-bit role. ### 3. Grafana datasource and correlation Add a `GrafanaDatasource` CR per `tracingStorages` entry in `templates/vtraces/grafana-datasource.yaml`, mirroring the logs datasource template and attaching through `instanceSelector: { matchLabels: { dashboards: grafana } }`. VictoriaTraces exposes a Jaeger-compatible query API, so the datasource is `type: jaeger` (or the dedicated VictoriaTraces datasource plugin, allow-listed like `victoriametrics-logs-datasource` is today) pointed at the `vtselect` service. Configure: - **Trace → logs**: link to the VictoriaLogs datasource keyed on `trace_id`. -- **Trace → metrics**: link to the VictoriaMetrics datasource for RED-style span metrics. +- **Trace → metrics**: link to the VictoriaMetrics datasource for RED-style span metrics. Grafana's trace-to-metrics only *links* to pre-existing metrics — it does not generate them (raised in review). So RED/span metrics need a source: the Stage-B collector's `spanmetrics` connector (recommended, produces RED metrics from spans and exports them to VictoriaMetrics), or native app instrumentation. Under Stage A (direct ingest, no collector) there are no span metrics — trace↔metrics correlation is therefore a Stage-B capability, and exemplars linking metrics→trace likewise depend on the metric producer emitting `trace_id` exemplars. - **Logs/metrics → trace**: derived fields on the existing datasources so a `trace_id` in a log or exemplar opens the trace. +**Read-side tenant isolation** (raised in review — the gap most likely to bite in central-backend mode): the write path attributes tenants via `AccountID`/`ProjectID`, but the *read* path must also scope a tenant's Grafana traces datasource to its own `AccountID`/`ProjectID` so tenant A cannot query tenant B's spans through the shared `vtselect`. This mirrors how logs/metrics rely on per-tenant labelling and/or per-tenant stacks; for traces on a shared backend it means the per-tenant `GrafanaDatasource` must carry the tenant's account/project scoping (or the tenant runs an isolated stack). This is called out explicitly in Open questions. + ### 4. Per-application opt-in Add a `tracing` struct to each participating app's `values.yaml` using the cozyvalues-gen annotation conventions, modelled on foundationdb's `monitoring.enabled` toggle and postgres's `backup` struct: @@ -171,28 +188,30 @@ flowchart LR ## Upgrade and rollback compatibility -The change is purely additive and opt-in. Existing clusters see no behavioural change until `tracingStorages` is set and an app flips `tracing.enabled`. An empty/absent `tracingStorages` renders no `VTCluster` (guarded, so it fails loudly only if a downstream component is told to expect one — matching the logs behaviour). No data migration is required. Rollback is removing the `tracingStorages` block and the per-app toggles; trace data in VictoriaTraces PVCs is discarded on backend removal (flagged: irreversible for already-stored spans, like logs). +The change is purely additive and opt-in. Existing clusters see no behavioural change until `tracingStorages` is set and an app flips `tracing.enabled`. An **absent** `tracingStorages` renders no `VTCluster` and succeeds (tracing stays off); an **explicitly empty** list fails the render per the absent-vs-empty contract. No data migration is required. Rollback is removing the `tracingStorages` block and the per-app toggles; trace data in VictoriaTraces PVCs is discarded on backend removal (flagged: irreversible for already-stored spans, like logs). ## Security -- **New tenant-supplied input**: the OTLP endpoint accepts spans from tenant workloads. In Stage A the shared `vtinsert` backend is the exposed surface (relying on VictoriaTraces' own limits and the per-tenant isolation of isolated stacks); Stage B's collector becomes the explicit trust boundary — enforcing per-tenant `tenant` resource attribution, rate-limits, and sampling so a noisy or hostile tenant cannot exhaust the shared backend. This hardening is the main reason to promote Stage B on a shared central backend. -- **Isolation**: central-backend mode keeps the existing tenant-labelling model; isolated mode (`packages/extra/monitoring`) keeps traces inside the tenant. +- **New tenant-supplied input**: the OTLP endpoint accepts spans from tenant workloads. In Stage A the shared `vtinsert` backend is the exposed surface (relying on VictoriaTraces' own limits and the per-tenant isolation of isolated stacks); Stage B's collector becomes the explicit trust boundary — enforcing per-tenant attribution, rate-limits, and sampling so a noisy or hostile tenant cannot exhaust the shared backend. This hardening is the main reason to promote Stage B on a shared central backend. +- **Tenant attribution must be trusted**: `AccountID`/`ProjectID` (and the `tenant` attribute) must be injected from authenticated workload identity, never accepted verbatim from a tenant that could spoof another's IDs. On the shared backend this injection belongs at a boundary the tenant cannot bypass — the per-tenant proxy or the collector — not purely in tenant-controlled app config. +- **Isolation (write and read)**: write-side attribution alone is not isolation. The **read** path must scope each tenant's Grafana traces datasource to its own `AccountID`/`ProjectID` so a shared `vtselect` cannot leak spans across tenants; central-backend mode keeps the existing tenant-labelling model, isolated mode (`packages/extra/monitoring`) keeps traces inside the tenant. - **Transport**: OTLP endpoints should be TLS-terminated; align with the unified TLS/PKI model (`design-proposals/unified-tls-pki`) rather than minting bespoke certs. - **RBAC**: new `VTCluster`/`GrafanaDatasource`/collector resources need the same narrowly-scoped RBAC the metrics/logs equivalents already have. No new secret classes are introduced beyond the OTLP endpoint credentials, if any. ## Failure and edge cases -- Empty `tracingStorages` while a consumer expects a backend → loud render failure (mirrors `cozystack/cozystack#3181`), never a silent span black hole. +- `tracingStorages` **absent/unset** → tracing disabled, template renders nothing and succeeds. `tracingStorages: []` **explicitly empty** → loud render failure (mirrors `cozystack/cozystack#3181`), never a silent span black hole. Both paths are asserted in tests. - OTLP endpoint unreachable from an app (backend or collector down) → the app's exporter drops/retries per OTLP defaults; the app itself does not crash and serving is unaffected. -- Backend storage exhausted → oldest traces evicted per `retentionPeriod`; ingest backpressures at the receiver (`vtinsert` in Stage A, the collector in Stage B), not the app. +- Storage full — **age vs disk retention are independent** (raised in review). `retentionPeriod` only prunes by age; it does *not* bound disk. Without a disk cap a full PVC blocks ingest *before* old traces are evicted. Set `vtstorage.retentionMaxDiskSpaceUsagePercent` (or `-retention.maxDiskSpaceUsageBytes`) via `retentionDiskSpaceUsage`, and add a PVC/disk-usage capacity alert. Covered by a PVC-exhaustion test. - `tracing.enabled: false` → no sidecar, no env, no CR: zero overhead. - App emits OTLP but no backend deployed → the ExternalName resolves to nothing and the exporter drops/retries harmlessly; the app toggle is documented to require a `tracingStorages` backend. +- App sets no `AccountID`/`ProjectID` → spans land in the default tenant `0:0` (visible to whoever can read that tenant). The write-boundary injection above must prevent this on a shared backend. ## Testing -- **Unit/lint**: `helm template` + `helm lint` for the new `vtraces` templates and each app's `tracing` block; assert `VTCluster`, the collector, and the `GrafanaDatasource` render only when configured, and that an empty `tracingStorages` fails the render. -- **e2e** (Chainsaw, per `docs/agents/e2e-testing.md`): deploy the monitoring stack with a `tracingStorages` entry, deploy one app (start with a native-OTLP engine, e.g. ClickHouse) with `tracing.enabled: true`, generate activity, then assert a trace is queryable via the `vtselect` Jaeger API and visible in Grafana. -- **Manual**: verify trace→logs and trace→metrics pivots in Grafana. +- **Unit/lint**: `helm template` + `helm lint` for the new `vtraces` templates and each app's `tracing` block; assert `VTCluster`, the collector, and the `GrafanaDatasource` render only when configured; assert the **absent-vs-empty** contract (absent `tracingStorages` → renders nothing and succeeds; `tracingStorages: []` → render fails); assert `retentionDiskSpaceUsage` maps to the disk-retention field when set. +- **e2e** (Chainsaw, per `docs/agents/e2e-testing.md`): deploy the monitoring stack with a `tracingStorages` entry, deploy one app (start with a native-OTLP engine, e.g. ClickHouse) with `tracing.enabled: true`, generate activity, then assert a trace is queryable via the `vtselect` query API and visible in Grafana; assert a second tenant cannot read the first tenant's spans (read-side isolation); a PVC-exhaustion case asserts ingest degrades safely with a disk cap set. +- **Manual**: verify trace→logs and (once the collector/spanmetrics land) trace→metrics pivots in Grafana. ## Rollout @@ -205,12 +224,15 @@ The change is purely additive and opt-in. Existing clusters see no behavioural c ## Open questions - **VictoriaTraces maturity**: the CRDs ship in the operator, but is VictoriaTraces production-ready at the version Cozystack pins? If not, Grafana Tempo is the drop-in fallback (see Alternatives) — the collector and per-app surfaces are backend-agnostic, so only the backend template and datasource type change. -- **`VTCluster` vs `VTSingle`** as the default: cluster for HA parity with metrics/logs, or single for a lighter footprint on small clusters? -- **Sampling**: head sampling at the app vs tail sampling at the collector; what platform default? (Only relevant once Stage B lands.) +- **`VTCluster` vs `VTSingle`**: lean toward supporting **both** (as metrics/logs do) — `VTSingle` for edge/dev/small clusters to cut overhead, `VTCluster` for production. Which is the default per stack? +- **Grafana traces datasource type (confirm before implementation)**: the design assumes `type: jaeger` against `vtselect` (VictoriaTraces' Jaeger-compatible query API), with the dedicated VictoriaTraces datasource plugin as the alternative. This is the one load-bearing claim not verifiable against the cozystack tree — the whole trace↔logs↔metrics UX depends on which actually lands, so it must be confirmed against the pinned VictoriaTraces version before building. +- **Read-side tenant isolation on a shared backend**: how exactly does the per-tenant Grafana datasource scope a tenant to its own `AccountID`/`ProjectID` on `vtselect`, so tenant A cannot read tenant B's spans? (Isolated per-tenant stacks avoid the question; the shared central backend does not.) +- **Sampling default**: head sampling at the app (`probabilistic_sampler`, scales freely) vs tail sampling at the collector (needs trace affinity — single replica or `loadbalancingexporter`)? Only relevant once Stage B lands. +- **Durability**: is single-instance `vtstorage` acceptable, or does the platform need cross-node span durability (collector replication into two independent backends)? - **External OTLP exposure**: should tenants be able to push spans from outside the cluster, and if so through which ingress/Gateway path? -- **Stage-B trigger**: what concrete signal (backend load, abuse, a tenant-attribution requirement) promotes the collector gateway from optional to default? +- **Stage-B trigger**: what concrete signal (backend load, abuse, the tenant-header requirement) promotes the collector gateway from optional to default? Note that shared-backend multi-tenancy effectively *needs* the collector (or a trusted proxy) to inject `AccountID`/`ProjectID`, so Stage B may be mandatory for a shared central backend rather than purely optional. -Resolved during design (recorded here so they are not re-litigated): ingest is **staged A→B** — direct-to-`vtinsert` first, collector gateway as an opt-in second — because `vtinsert` is OTLP-native and the ExternalName redirect keeps the app-facing endpoint stable across the switch. The collector is a gateway **Deployment**, not a DaemonSet, because OTLP is push-based (apps are the agents). Tracing opt-in lives in app `values.yaml`, not `WorkloadMonitor` (operational-only). +Resolved during design (recorded here so they are not re-litigated): ingest is **staged A→B** — direct-to-`vtinsert` first, collector gateway as an opt-in second — because `vtinsert` is OTLP-native and the ExternalName redirect keeps the app-facing endpoint stable across the switch. The collector is a gateway **Deployment**, not a DaemonSet, because OTLP is push-based (apps are the agents). Tail sampling, if used, runs single-replica or behind a `loadbalancingexporter` (`routing_key: traceID`) to stay trace-affine. The `tracingStorages` contract is absent→disabled-and-render-ok, explicit-empty→fail. Tracing opt-in lives in app `values.yaml`, not `WorkloadMonitor` (operational-only). ## Alternatives considered From 441b4723c654a16c0316ec9df35417e30e421840 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Sat, 18 Jul 2026 21:13:53 +0300 Subject: [PATCH 04/10] =?UTF-8?q?Move=20proposal=20status=20Draft=20?= =?UTF-8?q?=E2=86=92=20Review?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Alexey Artamonov --- design-proposals/distributed-tracing/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/design-proposals/distributed-tracing/README.md b/design-proposals/distributed-tracing/README.md index e9a7138..58c6db8 100644 --- a/design-proposals/distributed-tracing/README.md +++ b/design-proposals/distributed-tracing/README.md @@ -3,7 +3,7 @@ - **Title:** `Distributed tracing for managed applications via OTLP and VictoriaTraces` - **Author(s):** `@scooby87` - **Date:** `2026-07-16` -- **Status:** Draft +- **Status:** Review ## Overview From 65f5f0c7678a30a0ff251aa14dda6cedf2137cf5 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Mon, 20 Jul 2026 12:32:46 +0300 Subject: [PATCH 05/10] Address 2nd CodeRabbit pass: percent-only disk cap, stable gRPC contract, headers_setter - retentionDiskUsagePercent is percent-only (maps to retentionMaxDiskSpaceUsagePercent); drop the ambiguous bytes-or-percent field. - Fix the Stage A->B endpoint contradiction: fix the app-facing contract to a port-explicit OTLP/gRPC :4317 Service (bare ExternalName can't carry port/path); gRPC chosen because vtinsert and collector OTLP/HTTP paths differ. - Stage B tenant headers via headers_setter extension (from_context), not static otlphttp.headers which cannot derive per-tenant AccountID/ProjectID. Signed-off-by: Alexey Artamonov --- .../distributed-tracing/README.md | 28 +++++++++---------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/design-proposals/distributed-tracing/README.md b/design-proposals/distributed-tracing/README.md index 58c6db8..4478c2c 100644 --- a/design-proposals/distributed-tracing/README.md +++ b/design-proposals/distributed-tracing/README.md @@ -64,7 +64,7 @@ Add a `tracingStorages` list to the monitoring values, parallel to `metricsStora tracingStorages: - name: generic retentionPeriod: "14d" # age-based retention; default 14 days per the requirement - retentionDiskSpaceUsage: "" # optional disk-based cap (e.g. "80%" / bytes); see note below + retentionDiskUsagePercent: "" # optional disk-based cap, percent only (e.g. "80"); see note below storage: 10Gi storageClassName: "" replicaCount: 2 # component scaling, NOT data replication (see durability note) @@ -93,7 +93,7 @@ spec: vtselect: { replicaCount: {{ .replicaCount | default 2 }} } vtstorage: retentionPeriod: {{ .retentionPeriod | quote }} - {{- with .retentionDiskSpaceUsage }} + {{- with .retentionDiskUsagePercent }} retentionMaxDiskSpaceUsagePercent: {{ . | quote }} {{- end }} replicaCount: {{ .replicaCount | default 2 }} @@ -116,23 +116,23 @@ The monitoring HelmRelease must gate readiness on the new `VTCluster` exactly as ### 2. OTLP ingest (staged: direct-to-backend, then a collector gateway) -`vtinsert` accepts OTLP natively, so the ingest path mirrors logs one-for-one: where fluent-bit ships to `vlinsert-generic`, a traced application ships OTLP to `vtinsert-generic`. This proposal stages the ingest so the platform gets value immediately and grows into the collector the client asked for, with **no app-visible endpoint change between stages**. +`vtinsert` accepts OTLP natively, so the ingest path mirrors logs one-for-one: where fluent-bit ships to `vlinsert-generic`, a traced application ships OTLP to `vtinsert`. This proposal stages the ingest so the platform gets value immediately and grows into the collector the client asked for, behind **one stable app-facing endpoint** — an explicit in-cluster Service on OTLP/gRPC `4317` — so promoting Stage A→B swaps what that Service targets (backend → collector), not the app's configuration. -**Stage A (MVP) — direct to `vtinsert`.** Applications push OTLP straight to the VictoriaTraces insert service. Add an `otel-traces` (or `vtinsert-generic`) **ExternalName redirect** to `packages/system/cozystack-basics/templates/monitoring-external-services.yaml`, alongside the existing `vlinsert-generic`/`vminsert-*` entries, pointing `*.cozy-monitoring.svc` → `*.tenant-root.svc.cluster.local`, gated on the same `_cluster.monitoring-enabled` flag (from `monitoring.rootEnabled`). This is the exact mechanism logs and metrics already use; central-vs-isolated tenants work identically with zero new machinery. +**Stage A (MVP) — direct to `vtinsert`.** Applications push OTLP/gRPC to the stable Service `otel-traces.cozy-monitoring.svc:4317`, which in Stage A resolves to `vtinsert`'s OTLP/gRPC listener (enabled by setting `-otlpGRPCListenAddr` on the `VTCluster`). Tenant redirection to the central stack reuses the mechanism in `packages/system/cozystack-basics/templates/monitoring-external-services.yaml` (gated on `_cluster.monitoring-enabled` from `monitoring.rootEnabled`) that logs and metrics already use — but as a **port-explicit** Service, not a bare `ExternalName`. -**OTLP service contract** (raised in review — the exact ports/path matter): an `ExternalName` Service only aliases DNS; it does not remap ports or translate the OTLP/HTTP path. `vtinsert` defaults to OTLP/HTTP on port `10481` at path `/insert/opentelemetry/v1/traces`, and OTLP/gRPC only when `-otlpGRPCListenAddr` is set on the component (conventionally `4317`). So Stage A must be precise: either (a) point apps at `vtinsert`'s real OTLP/HTTP endpoint (`…:10481/insert/opentelemetry/v1/traces`) and enable the gRPC listener on the `VTCluster` if gRPC is wanted, redirecting via a `ClusterIP`/`ExternalName` Service that exposes those ports, or (b) front `vtinsert` with the Stage-B collector, which owns the canonical `4317/4318` OTLP listeners. The earlier "stable `…:4317`" shorthand assumes the collector or an explicit port-mapped Service — plain DNS aliasing alone is not enough. +**OTLP service contract** (raised in review — the exact ports/path matter, and a bare `ExternalName` cannot carry them): an `ExternalName` only aliases DNS; it does not remap ports or translate the OTLP/HTTP path. `vtinsert` defaults to OTLP/HTTP on port `10481` at path `/insert/opentelemetry/v1/traces`, and to OTLP/gRPC only when `-otlpGRPCListenAddr` is set (conventionally `4317`). Because the collector's OTLP/HTTP path (`/v1/traces`) differs from `vtinsert`'s, **OTLP/gRPC is the chosen stable app contract**: gRPC has no path, so the same `…:4317` Service works whether it fronts `vtinsert` (Stage A) or the collector (Stage B). The redirect is therefore an explicit ports-carrying Service (`ClusterIP` with an `externalName`-style upstream, or a small proxy), never plain DNS aliasing. OTLP/HTTP users must standardize on the collector from the start, since its ingest path is not interchangeable with `vtinsert`'s. -**Tenant routing needs headers** (raised in review — this is the load-bearing multi-tenancy point): VictoriaTraces attributes a tenant from the `AccountID`/`ProjectID` request headers (query-string equivalents take priority); with neither set, everything lands in the default tenant `0:0`. Neither an `ExternalName` redirect nor a collector `resource` processor sets those headers. So on a shared central backend the write path must inject `AccountID`/`ProjectID` derived from authenticated workload identity — from the app's OTLP exporter, a trusted per-tenant proxy, or (Stage B) the collector's `otlphttp` exporter `headers:` block — in addition to stamping the `tenant` resource attribute for query-time labelling. Isolated per-tenant stacks (`packages/extra/monitoring`) sidestep this by not sharing a backend. +**Tenant routing needs headers** (raised in review — this is the load-bearing multi-tenancy point): VictoriaTraces attributes a tenant from the `AccountID`/`ProjectID` request headers (query-string equivalents take priority); with neither set, everything lands in the default tenant `0:0`. Neither an `ExternalName` redirect nor a collector `resource` processor sets those headers. So on a shared central backend the write path must inject `AccountID`/`ProjectID` derived from authenticated workload identity — from the app's OTLP exporter, a trusted per-tenant proxy, or (Stage B) the collector's `headers_setter` extension (see below) — in addition to stamping the `tenant` resource attribute for query-time labelling. Isolated per-tenant stacks (`packages/extra/monitoring`) sidestep this by not sharing a backend. **Stage B — toggleable OpenTelemetry Collector gateway.** Add an OpenTelemetry Collector (gateway **Deployment** in `cozy-monitoring`, following the `packages/system/monitoring-agents` pattern) in front of `vtinsert` for the capabilities direct ingest can't provide: central sampling, rate-limiting, and per-`tenant` resource attribution. It is the "OpenTelemetry Collector/Agent via an option" the client explicitly requested, and it is opt-in. - Receivers: `otlp` on gRPC `4317` and HTTP `4318` (the collector owns these canonical ports; it translates to `vtinsert`'s `10481`/`/insert/opentelemetry/v1/traces` on export). - Processors: `batch`, a sampler (see the affinity note), and `resource` to stamp the `tenant` attribute. -- Exporter: `otlphttp` to the `vtinsert` service of the `tracingStorages` backend, with the `headers:` block setting `AccountID`/`ProjectID` from the authenticated tenant identity (not just the resource attribute). +- Exporter: `otlphttp`/`otlp` to the `vtinsert` service of the `tracingStorages` backend. Per-request `AccountID`/`ProjectID` must come from the `headers_setter` extension (`from_context`, with the OTLP receiver `include_metadata: true` and the `batch` processor preserving metadata) — **static `otlphttp.headers` cannot derive per-tenant IDs** (raised in review) and would route every tenant to one account. A trusted per-tenant proxy is the alternative; tenants must not be able to set their own routing headers. **Tail-sampling affinity** (raised in review): tail sampling must see *all* spans of a trace on one instance, but a plain Kubernetes Service round-robins spans across collector replicas and would make sampling decisions on incomplete traces. So the sampling tier must run at a single replica, or use the two-tier pattern — a first tier with the `loadbalancingexporter` (`routing_key: traceID`) consistently hashing each trace to a fixed second-tier instance that runs `tail_sampling`. Head sampling (`probabilistic_sampler`) has no such constraint and can scale freely; the platform default and this trade-off are an Open question. -Applications always target a stable in-cluster endpoint (e.g. `…cozy-monitoring.svc:4317` once the collector fronts ingest); flipping from Stage A to Stage B only re-points that ExternalName from `vtinsert` to the collector, so the migration is transparent to every traced app. A gateway **Deployment** (not a DaemonSet) is correct here because OTLP is push-based over the network — apps are the agents; the collector plays the centralized `vtinsert`/vmagent role, not the node-local fluent-bit role. +Because the app-facing contract is the fixed `otel-traces…:4317` gRPC Service, promoting Stage A→B changes only what that Service targets (`vtinsert` → collector) — a platform-side change, with no app reconfiguration (this holds for gRPC; OTLP/HTTP is not path-interchangeable, so HTTP users adopt the collector from the start). A gateway **Deployment** (not a DaemonSet) is correct here because OTLP is push-based over the network — apps are the agents; the collector plays the centralized `vtinsert`/vmagent role, not the node-local fluent-bit role. ### 3. Grafana datasource and correlation @@ -170,9 +170,9 @@ How an app emits spans depends on the engine: ```mermaid flowchart LR - app["Managed app
(tracing.enabled)"] -- "OTLP 4317/4318
(stable svc endpoint)" --> ext["ExternalName redirect
cozy-monitoring → tenant-root"] - ext -- "Stage A: direct" --> vt["VictoriaTraces
VTCluster (vtinsert→vtstorage)"] - ext -. "Stage B: via gateway" .-> col["OpenTelemetry Collector
(sampling / rate-limit / tenant)"] + app["Managed app
(tracing.enabled)"] -- "OTLP/gRPC :4317
(stable Service)" --> svc["otel-traces Service
cozy-monitoring (port-explicit)"] + svc -- "Stage A: → vtinsert gRPC" --> vt["VictoriaTraces
VTCluster (vtinsert→vtstorage)"] + svc -. "Stage B: → collector" .-> col["OpenTelemetry Collector
(sampling / rate-limit / AccountID+ProjectID)"] col -- OTLP --> vt gr["Grafana"] -- Jaeger query --> vt gr -. trace_id .-> vl["VictoriaLogs"] @@ -202,14 +202,14 @@ The change is purely additive and opt-in. Existing clusters see no behavioural c - `tracingStorages` **absent/unset** → tracing disabled, template renders nothing and succeeds. `tracingStorages: []` **explicitly empty** → loud render failure (mirrors `cozystack/cozystack#3181`), never a silent span black hole. Both paths are asserted in tests. - OTLP endpoint unreachable from an app (backend or collector down) → the app's exporter drops/retries per OTLP defaults; the app itself does not crash and serving is unaffected. -- Storage full — **age vs disk retention are independent** (raised in review). `retentionPeriod` only prunes by age; it does *not* bound disk. Without a disk cap a full PVC blocks ingest *before* old traces are evicted. Set `vtstorage.retentionMaxDiskSpaceUsagePercent` (or `-retention.maxDiskSpaceUsageBytes`) via `retentionDiskSpaceUsage`, and add a PVC/disk-usage capacity alert. Covered by a PVC-exhaustion test. +- Storage full — **age vs disk retention are independent** (raised in review). `retentionPeriod` only prunes by age; it does *not* bound disk. Without a disk cap a full PVC blocks ingest *before* old traces are evicted. Set `vtstorage.retentionMaxDiskSpaceUsagePercent` via the percent-only `retentionDiskUsagePercent` values field (upstream also offers a mutually-exclusive `-retention.maxDiskSpaceUsageBytes`, deliberately not exposed here to keep the field unambiguous), and add a PVC/disk-usage capacity alert. Covered by a PVC-exhaustion test. - `tracing.enabled: false` → no sidecar, no env, no CR: zero overhead. - App emits OTLP but no backend deployed → the ExternalName resolves to nothing and the exporter drops/retries harmlessly; the app toggle is documented to require a `tracingStorages` backend. - App sets no `AccountID`/`ProjectID` → spans land in the default tenant `0:0` (visible to whoever can read that tenant). The write-boundary injection above must prevent this on a shared backend. ## Testing -- **Unit/lint**: `helm template` + `helm lint` for the new `vtraces` templates and each app's `tracing` block; assert `VTCluster`, the collector, and the `GrafanaDatasource` render only when configured; assert the **absent-vs-empty** contract (absent `tracingStorages` → renders nothing and succeeds; `tracingStorages: []` → render fails); assert `retentionDiskSpaceUsage` maps to the disk-retention field when set. +- **Unit/lint**: `helm template` + `helm lint` for the new `vtraces` templates and each app's `tracing` block; assert `VTCluster`, the collector, and the `GrafanaDatasource` render only when configured; assert the **absent-vs-empty** contract (absent `tracingStorages` → renders nothing and succeeds; `tracingStorages: []` → render fails); assert `retentionDiskUsagePercent` maps to `retentionMaxDiskSpaceUsagePercent` when set. - **e2e** (Chainsaw, per `docs/agents/e2e-testing.md`): deploy the monitoring stack with a `tracingStorages` entry, deploy one app (start with a native-OTLP engine, e.g. ClickHouse) with `tracing.enabled: true`, generate activity, then assert a trace is queryable via the `vtselect` query API and visible in Grafana; assert a second tenant cannot read the first tenant's spans (read-side isolation); a PVC-exhaustion case asserts ingest degrades safely with a disk cap set. - **Manual**: verify trace→logs and (once the collector/spanmetrics land) trace→metrics pivots in Grafana. @@ -218,7 +218,7 @@ The change is purely additive and opt-in. Existing clusters see no behavioural c 1. **Stage A — backend + direct ingest**: `tracingStorages`/`VTCluster` in `packages/system/monitoring` (+ `packages/extra/monitoring`) with the poller readiness gate, plus the `vtinsert` ExternalName redirect in `packages/system/cozystack-basics`. Apps can push OTLP directly. No collector yet. 2. **Grafana**: traces datasource + correlation links. 3. **Per-app toggles**: start with native-OTLP engines (ClickHouse, NATS), then sidecar-based engines (Kafka, RabbitMQ, MariaDB, Redis, Postgres), one PR per app. -4. **Stage B — collector gateway**: add the toggleable OpenTelemetry Collector Deployment and re-point the ExternalName from `vtinsert` to the collector (transparent to apps). Ships sampling/rate-limiting/tenant-attribution. +4. **Stage B — collector gateway**: add the toggleable OpenTelemetry Collector Deployment and re-point the `otel-traces:4317` Service from `vtinsert` to the collector (no app change for gRPC clients). Ships sampling/rate-limiting and `AccountID`/`ProjectID` header injection via `headers_setter`. 5. **Docs**: enablement guide under `docs/observability/`. ## Open questions From f4d552c77adc0574a63a0c858cc181873f561ed9 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Mon, 20 Jul 2026 14:32:38 +0300 Subject: [PATCH 06/10] Correct VTCluster spec keys, disk-cap field, and topology per CRD verification Blocking fixes from /branch-review, verified against the pinned operator CRD: - VTCluster sub-spec keys are spec.insert/select/storage (prefix dropped), NOT vtinsert/vtselect/vtstorage; rendered workloads/services keep the vt prefix. - Disk cap field is spec.storage.retentionMaxDiskSpaceUsageBytes (bytes); no percent CR field exists. values field is now retentionDiskUsageBytes (bytes); earlier percent-only mapping was inverted. - Topology: monitoring installs into cozy-monitoring; ExternalName redirects live in cozy-monitoring and resolve to *.tenant-root.svc.cluster.local. - Nits: storageSize values key (avoid triple-storage stutter); WorkloadMonitor wording tightened + redis listed; confirm-before-impl caveat lists OTLP wire details + service-name prefix. Signed-off-by: Alexey Artamonov --- .../distributed-tracing/README.md | 34 +++++++++---------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/design-proposals/distributed-tracing/README.md b/design-proposals/distributed-tracing/README.md index 4478c2c..e0b8e4b 100644 --- a/design-proposals/distributed-tracing/README.md +++ b/design-proposals/distributed-tracing/README.md @@ -22,15 +22,15 @@ In scope: a traces backend (platform-wide and per-tenant), an OTLP ingest gatewa ## Context -Cozystack's observability is multi-tenant with a central backend. The platform stack runs in `tenant-root` (namespace `cozy-monitoring`) and hosts VictoriaMetrics, VictoriaLogs, Grafana (via grafana-operator), Alerta and vmalert. Tenants either ship signals to the central backend (ExternalName redirects in tenant namespaces point at the root stack; vmagent stamps a `tenant:` external label) or run their own isolated stack from `packages/extra/monitoring`. +Cozystack's observability is multi-tenant with a central backend. The `monitoring` component (VictoriaMetrics, VictoriaLogs, Grafana via grafana-operator, Alerta, vmalert) installs into the `cozy-monitoring` namespace (`packages/core/platform/sources/monitoring.yaml`), and its backing insert/select services resolve to `*.tenant-root.svc.cluster.local`. The redirect wiring lives in `cozy-monitoring`: `packages/system/cozystack-basics/templates/monitoring-external-services.yaml` creates `ExternalName` services *there* (`vlinsert-generic`, `vminsert-shortterm`, `vminsert-longterm`) whose `externalName` targets are in `tenant-root` — so `cozy-monitoring` holds the aliases and `tenant-root` holds what they resolve to, gated on `_cluster.monitoring-enabled`. vmagent stamps a `tenant:` external label; a tenant can instead run its own isolated stack from `packages/extra/monitoring`. Metrics storage is declared as a list of tiers in values and rendered into VictoriaMetrics CRs — `metricsStorages` in `packages/system/monitoring/values.yaml` defines a `shortterm` (3d) and a `longterm` (14d) tier. Logs storage follows the identical shape: `logsStorages` renders one `VLCluster` per entry in `packages/system/monitoring/templates/vlogs/vlogs.yaml`, with the retention period set on `vlstorage.retentionPeriod`, `managedMetadata` labels for application ownership, a label stamped on the storage PVC claim template so the post-delete cleanup hook can find it, and a load-bearing guard that **fails the render** when the list is empty rather than silently shipping to a non-existent endpoint (the fix for issue `cozystack/cozystack#3181`). Grafana datasources are provisioned as `GrafanaDatasource` CRs by grafana-operator, one per storage: `packages/system/monitoring/templates/vm/grafana-datasource.yaml` (type `prometheus`, per metrics tier) and `packages/system/monitoring/templates/vlogs/grafana-datasource.yaml` (type `victoriametrics-logs-datasource`, per logs storage). Every datasource attaches to Grafana through `instanceSelector: { matchLabels: { dashboards: grafana } }`. -Applications expose metrics today but there is no tracing surface. Most managed engines are scraped unconditionally through a `WorkloadMonitor` CR (clickhouse, kafka, rabbitmq, nats, mariadb) or a native operator mechanism (postgres/CNPG `enablePodMonitor`, redis via a `redis_exporter` sidecar + `VMServiceScrape`). The one app with an explicit observability toggle is foundationdb: `monitoring.enabled` in `packages/apps/foundationdb/values.yaml` gates whether its `WorkloadMonitor` renders — that toggle is the shape a `tracing.enabled` switch should copy. Note that `WorkloadMonitor` itself is not a fit for carrying tracing config: its controller (`internal/controller/workloadmonitor_controller.go`) reconciles it into `Workload` objects that track replicas/resources/operational status for the dashboard and billing surfaces — it does not emit scrape configs, and tracing opt-in therefore belongs in each app's `values.yaml`, not in `WorkloadMonitor`. +Applications expose metrics today but there is no tracing surface. Most managed engines declare a `WorkloadMonitor` CR (clickhouse, kafka, rabbitmq, nats, mariadb, redis) and/or a native operator mechanism (postgres/CNPG `enablePodMonitor`, redis's `redis_exporter` sidecar + `VMServiceScrape`). The one app with an explicit observability toggle is foundationdb: `monitoring.enabled` in `packages/apps/foundationdb/values.yaml` gates whether its `WorkloadMonitor` renders — that toggle is the shape a `tracing.enabled` switch should copy. Note that `WorkloadMonitor` itself is not a fit for carrying tracing config: its controller (`internal/controller/workloadmonitor_controller.go`) reconciles it into `Workload` objects that track replicas/resources/operational status (and query Prometheus for resource usage) for the dashboard and billing surfaces — the actual metric scrape configs are rendered by the individual app charts, not by this controller, so tracing opt-in belongs in each app's `values.yaml`, not in `WorkloadMonitor`. -Crucially, the tracing backend needs no new operator. The victoria-metrics-operator Cozystack already runs (appVersion `v0.68.4`, `packages/system/victoria-metrics-operator`) ships the VictoriaTraces CRDs `VTCluster` and `VTSingle` (`packages/system/victoria-metrics-operator/charts/victoria-metrics-operator/crd.yaml`, CRDs `vtclusters.operator.victoriametrics.com` and `vtsingles.operator.victoriametrics.com`). `VTCluster` decomposes into `VTInsert`/`VTStorage`/`VTSelect` — a direct analog of `VLCluster`'s `vlinsert`/`vlstorage`/`vlselect` — so the render template can be lifted from `vlogs.yaml` almost verbatim. +Crucially, the tracing backend needs no new operator. The victoria-metrics-operator Cozystack already runs (appVersion `v0.68.4`, `packages/system/victoria-metrics-operator`) ships the VictoriaTraces CRDs `VTCluster` and `VTSingle` (`packages/system/victoria-metrics-operator/charts/victoria-metrics-operator/crd.yaml`, CRDs `vtclusters.operator.victoriametrics.com` and `vtsingles.operator.victoriametrics.com`). `VTCluster` decomposes into VTInsert/VTStorage/VTSelect components — conceptually analogous to `VLCluster`'s insert/storage/select — but note the exact spec keys differ: `VLCluster` uses `vlinsert`/`vlselect`/`vlstorage`, whereas `VTCluster` **drops the prefix** and uses `spec.insert`/`spec.select`/`spec.storage` (verified against the CRD's printer columns `.spec.insert.replicaCount` etc.). The rendered workloads/services still carry the `vt` prefix (`vtinsert-*`/`vtselect-*`/`vtstorage-*`), same as `VLCluster` yields `vlinsert-*`. So the template is *shaped* like `vlogs.yaml` but the sub-spec keys are not a verbatim copy — this is the one place the analogy misleads, and it is corrected throughout below. ### The problem @@ -64,13 +64,13 @@ Add a `tracingStorages` list to the monitoring values, parallel to `metricsStora tracingStorages: - name: generic retentionPeriod: "14d" # age-based retention; default 14 days per the requirement - retentionDiskUsagePercent: "" # optional disk-based cap, percent only (e.g. "80"); see note below - storage: 10Gi + retentionDiskUsageBytes: "" # optional disk-based cap, byte quantity (e.g. "50GB"); see note below + storageSize: 10Gi # PVC size for vtstorage storageClassName: "" replicaCount: 2 # component scaling, NOT data replication (see durability note) ``` -Render one `VTCluster` per entry in a new `templates/vtraces/vtraces.yaml`, lifted from `templates/vlogs/vlogs.yaml`: `managedMetadata` labels for application ownership, configurable replica counts on `vtinsert`/`vtselect`/`vtstorage`, `vtstorage.retentionPeriod` from the entry, the storage-PVC claim-template label for the release-scoped post-delete cleanup hook, and the `cozystack/cozystack#3181` guard (see the absent-vs-empty contract below). +Render one `VTCluster` per entry in a new `templates/vtraces/vtraces.yaml`, shaped like `templates/vlogs/vlogs.yaml` (but with `VTCluster`'s prefix-less sub-spec keys): `managedMetadata` labels for application ownership, configurable replica counts on `spec.insert`/`spec.select`/`spec.storage`, `spec.storage.retentionPeriod` from the entry, the storage-PVC claim-template label for the release-scoped post-delete cleanup hook, and the `cozystack/cozystack#3181` guard (see the absent-vs-empty contract below). **Absent vs empty contract** (raised in review — the two must not be conflated): an **absent/unset** `tracingStorages` key means "tracing disabled", and the template renders nothing and succeeds (so tracing stays opt-in and a cluster that never wanted traces is never broken). An **explicitly empty list** (`tracingStorages: []`) is a misconfiguration — a consumer asked for tracing but declared no backend — and **fails the render**, exactly like the logs `#3181` guard. Both cases must be covered by tests. @@ -89,12 +89,12 @@ spec: apps.cozystack.io/application.group: apps.cozystack.io apps.cozystack.io/application.kind: Monitoring apps.cozystack.io/application.name: {{ $.Release.Name }} - vtinsert: { replicaCount: {{ .replicaCount | default 2 }} } - vtselect: { replicaCount: {{ .replicaCount | default 2 }} } - vtstorage: + insert: { replicaCount: {{ .replicaCount | default 2 }} } + select: { replicaCount: {{ .replicaCount | default 2 }} } + storage: retentionPeriod: {{ .retentionPeriod | quote }} - {{- with .retentionDiskUsagePercent }} - retentionMaxDiskSpaceUsagePercent: {{ . | quote }} + {{- with .retentionDiskUsageBytes }} + retentionMaxDiskSpaceUsageBytes: {{ . | quote }} {{- end }} replicaCount: {{ .replicaCount | default 2 }} storage: @@ -108,7 +108,7 @@ spec: {{- end }} resources: requests: - storage: {{ .storage }} + storage: {{ .storageSize }} {{- end }} ``` @@ -202,20 +202,20 @@ The change is purely additive and opt-in. Existing clusters see no behavioural c - `tracingStorages` **absent/unset** → tracing disabled, template renders nothing and succeeds. `tracingStorages: []` **explicitly empty** → loud render failure (mirrors `cozystack/cozystack#3181`), never a silent span black hole. Both paths are asserted in tests. - OTLP endpoint unreachable from an app (backend or collector down) → the app's exporter drops/retries per OTLP defaults; the app itself does not crash and serving is unaffected. -- Storage full — **age vs disk retention are independent** (raised in review). `retentionPeriod` only prunes by age; it does *not* bound disk. Without a disk cap a full PVC blocks ingest *before* old traces are evicted. Set `vtstorage.retentionMaxDiskSpaceUsagePercent` via the percent-only `retentionDiskUsagePercent` values field (upstream also offers a mutually-exclusive `-retention.maxDiskSpaceUsageBytes`, deliberately not exposed here to keep the field unambiguous), and add a PVC/disk-usage capacity alert. Covered by a PVC-exhaustion test. +- Storage full — **age vs disk retention are independent** (raised in review). `retentionPeriod` only prunes by age; it does *not* bound disk. Without a disk cap a full PVC blocks ingest *before* old traces are evicted. Set `spec.storage.retentionMaxDiskSpaceUsageBytes` (the disk-cap field the `VTCluster` CRD actually exposes — verified in the pinned operator; a percent form is *not* a CR field, only an upstream flag) via the byte-quantity `retentionDiskUsageBytes` values field, and add a PVC/disk-usage capacity alert. Covered by a PVC-exhaustion test. - `tracing.enabled: false` → no sidecar, no env, no CR: zero overhead. - App emits OTLP but no backend deployed → the ExternalName resolves to nothing and the exporter drops/retries harmlessly; the app toggle is documented to require a `tracingStorages` backend. - App sets no `AccountID`/`ProjectID` → spans land in the default tenant `0:0` (visible to whoever can read that tenant). The write-boundary injection above must prevent this on a shared backend. ## Testing -- **Unit/lint**: `helm template` + `helm lint` for the new `vtraces` templates and each app's `tracing` block; assert `VTCluster`, the collector, and the `GrafanaDatasource` render only when configured; assert the **absent-vs-empty** contract (absent `tracingStorages` → renders nothing and succeeds; `tracingStorages: []` → render fails); assert `retentionDiskUsagePercent` maps to `retentionMaxDiskSpaceUsagePercent` when set. +- **Unit/lint**: `helm template` + `helm lint` for the new `vtraces` templates and each app's `tracing` block; assert `VTCluster`, the collector, and the `GrafanaDatasource` render only when configured; assert the **absent-vs-empty** contract (absent `tracingStorages` → renders nothing and succeeds; `tracingStorages: []` → render fails); assert `retentionDiskUsageBytes` maps to `spec.storage.retentionMaxDiskSpaceUsageBytes` when set. - **e2e** (Chainsaw, per `docs/agents/e2e-testing.md`): deploy the monitoring stack with a `tracingStorages` entry, deploy one app (start with a native-OTLP engine, e.g. ClickHouse) with `tracing.enabled: true`, generate activity, then assert a trace is queryable via the `vtselect` query API and visible in Grafana; assert a second tenant cannot read the first tenant's spans (read-side isolation); a PVC-exhaustion case asserts ingest degrades safely with a disk cap set. - **Manual**: verify trace→logs and (once the collector/spanmetrics land) trace→metrics pivots in Grafana. ## Rollout -1. **Stage A — backend + direct ingest**: `tracingStorages`/`VTCluster` in `packages/system/monitoring` (+ `packages/extra/monitoring`) with the poller readiness gate, plus the `vtinsert` ExternalName redirect in `packages/system/cozystack-basics`. Apps can push OTLP directly. No collector yet. +1. **Stage A — backend + direct ingest**: `tracingStorages`/`VTCluster` in `packages/system/monitoring` (+ `packages/extra/monitoring`) with the poller readiness gate, plus the port-explicit `otel-traces:4317` gRPC redirect Service in `packages/system/cozystack-basics` (targeting `vtinsert`'s gRPC listener). Apps can push OTLP/gRPC directly. No collector yet. 2. **Grafana**: traces datasource + correlation links. 3. **Per-app toggles**: start with native-OTLP engines (ClickHouse, NATS), then sidecar-based engines (Kafka, RabbitMQ, MariaDB, Redis, Postgres), one PR per app. 4. **Stage B — collector gateway**: add the toggleable OpenTelemetry Collector Deployment and re-point the `otel-traces:4317` Service from `vtinsert` to the collector (no app change for gRPC clients). Ships sampling/rate-limiting and `AccountID`/`ProjectID` header injection via `headers_setter`. @@ -225,14 +225,14 @@ The change is purely additive and opt-in. Existing clusters see no behavioural c - **VictoriaTraces maturity**: the CRDs ship in the operator, but is VictoriaTraces production-ready at the version Cozystack pins? If not, Grafana Tempo is the drop-in fallback (see Alternatives) — the collector and per-app surfaces are backend-agnostic, so only the backend template and datasource type change. - **`VTCluster` vs `VTSingle`**: lean toward supporting **both** (as metrics/logs do) — `VTSingle` for edge/dev/small clusters to cut overhead, `VTCluster` for production. Which is the default per stack? -- **Grafana traces datasource type (confirm before implementation)**: the design assumes `type: jaeger` against `vtselect` (VictoriaTraces' Jaeger-compatible query API), with the dedicated VictoriaTraces datasource plugin as the alternative. This is the one load-bearing claim not verifiable against the cozystack tree — the whole trace↔logs↔metrics UX depends on which actually lands, so it must be confirmed against the pinned VictoriaTraces version before building. +- **Confirm-before-implementation (upstream VictoriaTraces surface)**: the `VTCluster` spec keys (`insert`/`select`/`storage`) and the disk-cap field (`retentionMaxDiskSpaceUsageBytes`) are verified against the pinned operator CRD. Still taken on upstream faith and to be confirmed against the pinned VictoriaTraces version before building: the Grafana datasource type (`type: jaeger` against `vtselect`, or the dedicated VictoriaTraces plugin — the whole correlation UX depends on which lands); the OTLP wire details (`vtinsert` HTTP `:10481` + `/insert/opentelemetry/v1/traces`, gRPC via `-otlpGRPCListenAddr`); and the exact rendered service-name prefix (`vtinsert-*`/`vtselect-*` vs `insert-*`). - **Read-side tenant isolation on a shared backend**: how exactly does the per-tenant Grafana datasource scope a tenant to its own `AccountID`/`ProjectID` on `vtselect`, so tenant A cannot read tenant B's spans? (Isolated per-tenant stacks avoid the question; the shared central backend does not.) - **Sampling default**: head sampling at the app (`probabilistic_sampler`, scales freely) vs tail sampling at the collector (needs trace affinity — single replica or `loadbalancingexporter`)? Only relevant once Stage B lands. - **Durability**: is single-instance `vtstorage` acceptable, or does the platform need cross-node span durability (collector replication into two independent backends)? - **External OTLP exposure**: should tenants be able to push spans from outside the cluster, and if so through which ingress/Gateway path? - **Stage-B trigger**: what concrete signal (backend load, abuse, the tenant-header requirement) promotes the collector gateway from optional to default? Note that shared-backend multi-tenancy effectively *needs* the collector (or a trusted proxy) to inject `AccountID`/`ProjectID`, so Stage B may be mandatory for a shared central backend rather than purely optional. -Resolved during design (recorded here so they are not re-litigated): ingest is **staged A→B** — direct-to-`vtinsert` first, collector gateway as an opt-in second — because `vtinsert` is OTLP-native and the ExternalName redirect keeps the app-facing endpoint stable across the switch. The collector is a gateway **Deployment**, not a DaemonSet, because OTLP is push-based (apps are the agents). Tail sampling, if used, runs single-replica or behind a `loadbalancingexporter` (`routing_key: traceID`) to stay trace-affine. The `tracingStorages` contract is absent→disabled-and-render-ok, explicit-empty→fail. Tracing opt-in lives in app `values.yaml`, not `WorkloadMonitor` (operational-only). +Resolved during design (recorded here so they are not re-litigated): ingest is **staged A→B** — direct-to-`vtinsert` first, collector gateway as an opt-in second — because `vtinsert` is OTLP-native and a **port-explicit OTLP/gRPC `:4317` Service** (not a bare `ExternalName`) keeps the app-facing endpoint stable across the switch. The collector is a gateway **Deployment**, not a DaemonSet, because OTLP is push-based (apps are the agents). Tail sampling, if used, runs single-replica or behind a `loadbalancingexporter` (`routing_key: traceID`) to stay trace-affine. The `tracingStorages` contract is absent→disabled-and-render-ok, explicit-empty→fail. Tracing opt-in lives in app `values.yaml`, not `WorkloadMonitor` (operational-only). ## Alternatives considered From dd44d1cccf7d2f8180dfd76ff8239f6ff8ea6045 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Mon, 20 Jul 2026 15:59:01 +0300 Subject: [PATCH 07/10] docs: drop 'raised in review' meta-annotations from the proposal Keep the substantive points; remove the review-attribution asides so the document reads as a standalone design rather than a review-response artifact. Signed-off-by: Alexey Artamonov --- design-proposals/distributed-tracing/README.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/design-proposals/distributed-tracing/README.md b/design-proposals/distributed-tracing/README.md index e0b8e4b..d2d9b8f 100644 --- a/design-proposals/distributed-tracing/README.md +++ b/design-proposals/distributed-tracing/README.md @@ -72,9 +72,9 @@ tracingStorages: Render one `VTCluster` per entry in a new `templates/vtraces/vtraces.yaml`, shaped like `templates/vlogs/vlogs.yaml` (but with `VTCluster`'s prefix-less sub-spec keys): `managedMetadata` labels for application ownership, configurable replica counts on `spec.insert`/`spec.select`/`spec.storage`, `spec.storage.retentionPeriod` from the entry, the storage-PVC claim-template label for the release-scoped post-delete cleanup hook, and the `cozystack/cozystack#3181` guard (see the absent-vs-empty contract below). -**Absent vs empty contract** (raised in review — the two must not be conflated): an **absent/unset** `tracingStorages` key means "tracing disabled", and the template renders nothing and succeeds (so tracing stays opt-in and a cluster that never wanted traces is never broken). An **explicitly empty list** (`tracingStorages: []`) is a misconfiguration — a consumer asked for tracing but declared no backend — and **fails the render**, exactly like the logs `#3181` guard. Both cases must be covered by tests. +**Absent vs empty contract** (the two must not be conflated): an **absent/unset** `tracingStorages` key means "tracing disabled", and the template renders nothing and succeeds (so tracing stays opt-in and a cluster that never wanted traces is never broken). An **explicitly empty list** (`tracingStorages: []`) is a misconfiguration — a consumer asked for tracing but declared no backend — and **fails the render**, exactly like the logs `#3181` guard. Both cases must be covered by tests. -**Durability note** (raised in review): `replicaCount` scales the number of `vtstorage` pods for throughput/availability of the *component*, but VictoriaTraces cluster mode does **not** replicate span data between storage nodes — losing a storage node can make some queries return partial results. This is the same durability model as `VLCluster`, and it is *not* HA data replication. Real cross-node durability requires either replication through Collectors into two independent VictoriaTraces backends or an equivalent design; this is called out in Open questions rather than solved here. +**Durability note:** `replicaCount` scales the number of `vtstorage` pods for throughput/availability of the *component*, but VictoriaTraces cluster mode does **not** replicate span data between storage nodes — losing a storage node can make some queries return partial results. This is the same durability model as `VLCluster`, and it is *not* HA data replication. Real cross-node durability requires either replication through Collectors into two independent VictoriaTraces backends or an equivalent design; this is called out in Open questions rather than solved here. ```yaml {{- range .Values.tracingStorages }} @@ -120,17 +120,17 @@ The monitoring HelmRelease must gate readiness on the new `VTCluster` exactly as **Stage A (MVP) — direct to `vtinsert`.** Applications push OTLP/gRPC to the stable Service `otel-traces.cozy-monitoring.svc:4317`, which in Stage A resolves to `vtinsert`'s OTLP/gRPC listener (enabled by setting `-otlpGRPCListenAddr` on the `VTCluster`). Tenant redirection to the central stack reuses the mechanism in `packages/system/cozystack-basics/templates/monitoring-external-services.yaml` (gated on `_cluster.monitoring-enabled` from `monitoring.rootEnabled`) that logs and metrics already use — but as a **port-explicit** Service, not a bare `ExternalName`. -**OTLP service contract** (raised in review — the exact ports/path matter, and a bare `ExternalName` cannot carry them): an `ExternalName` only aliases DNS; it does not remap ports or translate the OTLP/HTTP path. `vtinsert` defaults to OTLP/HTTP on port `10481` at path `/insert/opentelemetry/v1/traces`, and to OTLP/gRPC only when `-otlpGRPCListenAddr` is set (conventionally `4317`). Because the collector's OTLP/HTTP path (`/v1/traces`) differs from `vtinsert`'s, **OTLP/gRPC is the chosen stable app contract**: gRPC has no path, so the same `…:4317` Service works whether it fronts `vtinsert` (Stage A) or the collector (Stage B). The redirect is therefore an explicit ports-carrying Service (`ClusterIP` with an `externalName`-style upstream, or a small proxy), never plain DNS aliasing. OTLP/HTTP users must standardize on the collector from the start, since its ingest path is not interchangeable with `vtinsert`'s. +**OTLP service contract** (the exact ports/path matter, and a bare `ExternalName` cannot carry them): an `ExternalName` only aliases DNS; it does not remap ports or translate the OTLP/HTTP path. `vtinsert` defaults to OTLP/HTTP on port `10481` at path `/insert/opentelemetry/v1/traces`, and to OTLP/gRPC only when `-otlpGRPCListenAddr` is set (conventionally `4317`). Because the collector's OTLP/HTTP path (`/v1/traces`) differs from `vtinsert`'s, **OTLP/gRPC is the chosen stable app contract**: gRPC has no path, so the same `…:4317` Service works whether it fronts `vtinsert` (Stage A) or the collector (Stage B). The redirect is therefore an explicit ports-carrying Service (`ClusterIP` with an `externalName`-style upstream, or a small proxy), never plain DNS aliasing. OTLP/HTTP users must standardize on the collector from the start, since its ingest path is not interchangeable with `vtinsert`'s. -**Tenant routing needs headers** (raised in review — this is the load-bearing multi-tenancy point): VictoriaTraces attributes a tenant from the `AccountID`/`ProjectID` request headers (query-string equivalents take priority); with neither set, everything lands in the default tenant `0:0`. Neither an `ExternalName` redirect nor a collector `resource` processor sets those headers. So on a shared central backend the write path must inject `AccountID`/`ProjectID` derived from authenticated workload identity — from the app's OTLP exporter, a trusted per-tenant proxy, or (Stage B) the collector's `headers_setter` extension (see below) — in addition to stamping the `tenant` resource attribute for query-time labelling. Isolated per-tenant stacks (`packages/extra/monitoring`) sidestep this by not sharing a backend. +**Tenant routing needs headers** (the load-bearing multi-tenancy point): VictoriaTraces attributes a tenant from the `AccountID`/`ProjectID` request headers (query-string equivalents take priority); with neither set, everything lands in the default tenant `0:0`. Neither an `ExternalName` redirect nor a collector `resource` processor sets those headers. So on a shared central backend the write path must inject `AccountID`/`ProjectID` derived from authenticated workload identity — from the app's OTLP exporter, a trusted per-tenant proxy, or (Stage B) the collector's `headers_setter` extension (see below) — in addition to stamping the `tenant` resource attribute for query-time labelling. Isolated per-tenant stacks (`packages/extra/monitoring`) sidestep this by not sharing a backend. **Stage B — toggleable OpenTelemetry Collector gateway.** Add an OpenTelemetry Collector (gateway **Deployment** in `cozy-monitoring`, following the `packages/system/monitoring-agents` pattern) in front of `vtinsert` for the capabilities direct ingest can't provide: central sampling, rate-limiting, and per-`tenant` resource attribution. It is the "OpenTelemetry Collector/Agent via an option" the client explicitly requested, and it is opt-in. - Receivers: `otlp` on gRPC `4317` and HTTP `4318` (the collector owns these canonical ports; it translates to `vtinsert`'s `10481`/`/insert/opentelemetry/v1/traces` on export). - Processors: `batch`, a sampler (see the affinity note), and `resource` to stamp the `tenant` attribute. -- Exporter: `otlphttp`/`otlp` to the `vtinsert` service of the `tracingStorages` backend. Per-request `AccountID`/`ProjectID` must come from the `headers_setter` extension (`from_context`, with the OTLP receiver `include_metadata: true` and the `batch` processor preserving metadata) — **static `otlphttp.headers` cannot derive per-tenant IDs** (raised in review) and would route every tenant to one account. A trusted per-tenant proxy is the alternative; tenants must not be able to set their own routing headers. +- Exporter: `otlphttp`/`otlp` to the `vtinsert` service of the `tracingStorages` backend. Per-request `AccountID`/`ProjectID` must come from the `headers_setter` extension (`from_context`, with the OTLP receiver `include_metadata: true` and the `batch` processor preserving metadata) — **static `otlphttp.headers` cannot derive per-tenant IDs** and would route every tenant to one account. A trusted per-tenant proxy is the alternative; tenants must not be able to set their own routing headers. -**Tail-sampling affinity** (raised in review): tail sampling must see *all* spans of a trace on one instance, but a plain Kubernetes Service round-robins spans across collector replicas and would make sampling decisions on incomplete traces. So the sampling tier must run at a single replica, or use the two-tier pattern — a first tier with the `loadbalancingexporter` (`routing_key: traceID`) consistently hashing each trace to a fixed second-tier instance that runs `tail_sampling`. Head sampling (`probabilistic_sampler`) has no such constraint and can scale freely; the platform default and this trade-off are an Open question. +**Tail-sampling affinity:** tail sampling must see *all* spans of a trace on one instance, but a plain Kubernetes Service round-robins spans across collector replicas and would make sampling decisions on incomplete traces. So the sampling tier must run at a single replica, or use the two-tier pattern — a first tier with the `loadbalancingexporter` (`routing_key: traceID`) consistently hashing each trace to a fixed second-tier instance that runs `tail_sampling`. Head sampling (`probabilistic_sampler`) has no such constraint and can scale freely; the platform default and this trade-off are an Open question. Because the app-facing contract is the fixed `otel-traces…:4317` gRPC Service, promoting Stage A→B changes only what that Service targets (`vtinsert` → collector) — a platform-side change, with no app reconfiguration (this holds for gRPC; OTLP/HTTP is not path-interchangeable, so HTTP users adopt the collector from the start). A gateway **Deployment** (not a DaemonSet) is correct here because OTLP is push-based over the network — apps are the agents; the collector plays the centralized `vtinsert`/vmagent role, not the node-local fluent-bit role. @@ -139,10 +139,10 @@ Because the app-facing contract is the fixed `otel-traces…:4317` gRPC Service, Add a `GrafanaDatasource` CR per `tracingStorages` entry in `templates/vtraces/grafana-datasource.yaml`, mirroring the logs datasource template and attaching through `instanceSelector: { matchLabels: { dashboards: grafana } }`. VictoriaTraces exposes a Jaeger-compatible query API, so the datasource is `type: jaeger` (or the dedicated VictoriaTraces datasource plugin, allow-listed like `victoriametrics-logs-datasource` is today) pointed at the `vtselect` service. Configure: - **Trace → logs**: link to the VictoriaLogs datasource keyed on `trace_id`. -- **Trace → metrics**: link to the VictoriaMetrics datasource for RED-style span metrics. Grafana's trace-to-metrics only *links* to pre-existing metrics — it does not generate them (raised in review). So RED/span metrics need a source: the Stage-B collector's `spanmetrics` connector (recommended, produces RED metrics from spans and exports them to VictoriaMetrics), or native app instrumentation. Under Stage A (direct ingest, no collector) there are no span metrics — trace↔metrics correlation is therefore a Stage-B capability, and exemplars linking metrics→trace likewise depend on the metric producer emitting `trace_id` exemplars. +- **Trace → metrics**: link to the VictoriaMetrics datasource for RED-style span metrics. Grafana's trace-to-metrics only *links* to pre-existing metrics — it does not generate them. So RED/span metrics need a source: the Stage-B collector's `spanmetrics` connector (recommended, produces RED metrics from spans and exports them to VictoriaMetrics), or native app instrumentation. Under Stage A (direct ingest, no collector) there are no span metrics — trace↔metrics correlation is therefore a Stage-B capability, and exemplars linking metrics→trace likewise depend on the metric producer emitting `trace_id` exemplars. - **Logs/metrics → trace**: derived fields on the existing datasources so a `trace_id` in a log or exemplar opens the trace. -**Read-side tenant isolation** (raised in review — the gap most likely to bite in central-backend mode): the write path attributes tenants via `AccountID`/`ProjectID`, but the *read* path must also scope a tenant's Grafana traces datasource to its own `AccountID`/`ProjectID` so tenant A cannot query tenant B's spans through the shared `vtselect`. This mirrors how logs/metrics rely on per-tenant labelling and/or per-tenant stacks; for traces on a shared backend it means the per-tenant `GrafanaDatasource` must carry the tenant's account/project scoping (or the tenant runs an isolated stack). This is called out explicitly in Open questions. +**Read-side tenant isolation** (the gap most likely to bite in central-backend mode): the write path attributes tenants via `AccountID`/`ProjectID`, but the *read* path must also scope a tenant's Grafana traces datasource to its own `AccountID`/`ProjectID` so tenant A cannot query tenant B's spans through the shared `vtselect`. This mirrors how logs/metrics rely on per-tenant labelling and/or per-tenant stacks; for traces on a shared backend it means the per-tenant `GrafanaDatasource` must carry the tenant's account/project scoping (or the tenant runs an isolated stack). This is called out explicitly in Open questions. ### 4. Per-application opt-in @@ -202,7 +202,7 @@ The change is purely additive and opt-in. Existing clusters see no behavioural c - `tracingStorages` **absent/unset** → tracing disabled, template renders nothing and succeeds. `tracingStorages: []` **explicitly empty** → loud render failure (mirrors `cozystack/cozystack#3181`), never a silent span black hole. Both paths are asserted in tests. - OTLP endpoint unreachable from an app (backend or collector down) → the app's exporter drops/retries per OTLP defaults; the app itself does not crash and serving is unaffected. -- Storage full — **age vs disk retention are independent** (raised in review). `retentionPeriod` only prunes by age; it does *not* bound disk. Without a disk cap a full PVC blocks ingest *before* old traces are evicted. Set `spec.storage.retentionMaxDiskSpaceUsageBytes` (the disk-cap field the `VTCluster` CRD actually exposes — verified in the pinned operator; a percent form is *not* a CR field, only an upstream flag) via the byte-quantity `retentionDiskUsageBytes` values field, and add a PVC/disk-usage capacity alert. Covered by a PVC-exhaustion test. +- Storage full — **age vs disk retention are independent**. `retentionPeriod` only prunes by age; it does *not* bound disk. Without a disk cap a full PVC blocks ingest *before* old traces are evicted. Set `spec.storage.retentionMaxDiskSpaceUsageBytes` (the disk-cap field the `VTCluster` CRD actually exposes — verified in the pinned operator; a percent form is *not* a CR field, only an upstream flag) via the byte-quantity `retentionDiskUsageBytes` values field, and add a PVC/disk-usage capacity alert. Covered by a PVC-exhaustion test. - `tracing.enabled: false` → no sidecar, no env, no CR: zero overhead. - App emits OTLP but no backend deployed → the ExternalName resolves to nothing and the exporter drops/retries harmlessly; the app toggle is documented to require a `tracingStorages` backend. - App sets no `AccountID`/`ProjectID` → spans land in the default tenant `0:0` (visible to whoever can read that tenant). The write-boundary injection above must prevent this on a shared backend. From 7c9394cbd90b46659eb935477c39d3cefad8a404 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Mon, 20 Jul 2026 18:55:58 +0300 Subject: [PATCH 08/10] Address 2026-07-20 review: drop unverified query-string tenant precedence Signed-off-by: Alexey Artamonov --- design-proposals/distributed-tracing/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/design-proposals/distributed-tracing/README.md b/design-proposals/distributed-tracing/README.md index d2d9b8f..d304ad2 100644 --- a/design-proposals/distributed-tracing/README.md +++ b/design-proposals/distributed-tracing/README.md @@ -122,7 +122,7 @@ The monitoring HelmRelease must gate readiness on the new `VTCluster` exactly as **OTLP service contract** (the exact ports/path matter, and a bare `ExternalName` cannot carry them): an `ExternalName` only aliases DNS; it does not remap ports or translate the OTLP/HTTP path. `vtinsert` defaults to OTLP/HTTP on port `10481` at path `/insert/opentelemetry/v1/traces`, and to OTLP/gRPC only when `-otlpGRPCListenAddr` is set (conventionally `4317`). Because the collector's OTLP/HTTP path (`/v1/traces`) differs from `vtinsert`'s, **OTLP/gRPC is the chosen stable app contract**: gRPC has no path, so the same `…:4317` Service works whether it fronts `vtinsert` (Stage A) or the collector (Stage B). The redirect is therefore an explicit ports-carrying Service (`ClusterIP` with an `externalName`-style upstream, or a small proxy), never plain DNS aliasing. OTLP/HTTP users must standardize on the collector from the start, since its ingest path is not interchangeable with `vtinsert`'s. -**Tenant routing needs headers** (the load-bearing multi-tenancy point): VictoriaTraces attributes a tenant from the `AccountID`/`ProjectID` request headers (query-string equivalents take priority); with neither set, everything lands in the default tenant `0:0`. Neither an `ExternalName` redirect nor a collector `resource` processor sets those headers. So on a shared central backend the write path must inject `AccountID`/`ProjectID` derived from authenticated workload identity — from the app's OTLP exporter, a trusted per-tenant proxy, or (Stage B) the collector's `headers_setter` extension (see below) — in addition to stamping the `tenant` resource attribute for query-time labelling. Isolated per-tenant stacks (`packages/extra/monitoring`) sidestep this by not sharing a backend. +**Tenant routing needs headers** (the load-bearing multi-tenancy point): VictoriaTraces attributes a tenant from the `AccountID`/`ProjectID` request headers; with neither set, everything lands in the default tenant `0:0`. Neither an `ExternalName` redirect nor a collector `resource` processor sets those headers. So on a shared central backend the write path must inject `AccountID`/`ProjectID` derived from authenticated workload identity — from the app's OTLP exporter, a trusted per-tenant proxy, or (Stage B) the collector's `headers_setter` extension (see below) — in addition to stamping the `tenant` resource attribute for query-time labelling. Isolated per-tenant stacks (`packages/extra/monitoring`) sidestep this by not sharing a backend. **Stage B — toggleable OpenTelemetry Collector gateway.** Add an OpenTelemetry Collector (gateway **Deployment** in `cozy-monitoring`, following the `packages/system/monitoring-agents` pattern) in front of `vtinsert` for the capabilities direct ingest can't provide: central sampling, rate-limiting, and per-`tenant` resource attribution. It is the "OpenTelemetry Collector/Agent via an option" the client explicitly requested, and it is opt-in. From 3cb278cd76f6650bed2166cba46466dce2f6aa07 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Mon, 20 Jul 2026 19:25:38 +0300 Subject: [PATCH 09/10] Address branch review: resolve read-side isolation, VTSingle mode, fix ExternalName wording - Resolve read-side tenant isolation open question: structural isolation (isolated per-tenant stack + same-namespace vtselect datasource + NetworkPolicy, account 0), matching logs/metrics; vtselect has no authz so a datasource header pin is routing, not a security boundary; shared backend needs vmauth (deferred). Reconcile Grafana + Security sections. - Resolve VTCluster-vs-VTSingle: per-entry mode: cluster|single (default cluster). - Fix contradiction on the no-backend failure mode: Service has no backing endpoints, not 'ExternalName resolves to nothing' (design uses a port-explicit Service, not a bare ExternalName). - Soften the vtinsert-* service-name prefix to confirm-before-implementation. - Align write-path AccountID/ProjectID framing with the current account-0 + tenant-label convention. - Add revised-date suffix. Signed-off-by: Alexey Artamonov --- .../distributed-tracing/README.md | 22 +++++++++---------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/design-proposals/distributed-tracing/README.md b/design-proposals/distributed-tracing/README.md index d304ad2..996eac8 100644 --- a/design-proposals/distributed-tracing/README.md +++ b/design-proposals/distributed-tracing/README.md @@ -2,7 +2,7 @@ - **Title:** `Distributed tracing for managed applications via OTLP and VictoriaTraces` - **Author(s):** `@scooby87` -- **Date:** `2026-07-16` +- **Date:** `2026-07-16` (revised 2026-07-20) - **Status:** Review ## Overview @@ -30,7 +30,7 @@ Grafana datasources are provisioned as `GrafanaDatasource` CRs by grafana-operat Applications expose metrics today but there is no tracing surface. Most managed engines declare a `WorkloadMonitor` CR (clickhouse, kafka, rabbitmq, nats, mariadb, redis) and/or a native operator mechanism (postgres/CNPG `enablePodMonitor`, redis's `redis_exporter` sidecar + `VMServiceScrape`). The one app with an explicit observability toggle is foundationdb: `monitoring.enabled` in `packages/apps/foundationdb/values.yaml` gates whether its `WorkloadMonitor` renders — that toggle is the shape a `tracing.enabled` switch should copy. Note that `WorkloadMonitor` itself is not a fit for carrying tracing config: its controller (`internal/controller/workloadmonitor_controller.go`) reconciles it into `Workload` objects that track replicas/resources/operational status (and query Prometheus for resource usage) for the dashboard and billing surfaces — the actual metric scrape configs are rendered by the individual app charts, not by this controller, so tracing opt-in belongs in each app's `values.yaml`, not in `WorkloadMonitor`. -Crucially, the tracing backend needs no new operator. The victoria-metrics-operator Cozystack already runs (appVersion `v0.68.4`, `packages/system/victoria-metrics-operator`) ships the VictoriaTraces CRDs `VTCluster` and `VTSingle` (`packages/system/victoria-metrics-operator/charts/victoria-metrics-operator/crd.yaml`, CRDs `vtclusters.operator.victoriametrics.com` and `vtsingles.operator.victoriametrics.com`). `VTCluster` decomposes into VTInsert/VTStorage/VTSelect components — conceptually analogous to `VLCluster`'s insert/storage/select — but note the exact spec keys differ: `VLCluster` uses `vlinsert`/`vlselect`/`vlstorage`, whereas `VTCluster` **drops the prefix** and uses `spec.insert`/`spec.select`/`spec.storage` (verified against the CRD's printer columns `.spec.insert.replicaCount` etc.). The rendered workloads/services still carry the `vt` prefix (`vtinsert-*`/`vtselect-*`/`vtstorage-*`), same as `VLCluster` yields `vlinsert-*`. So the template is *shaped* like `vlogs.yaml` but the sub-spec keys are not a verbatim copy — this is the one place the analogy misleads, and it is corrected throughout below. +Crucially, the tracing backend needs no new operator. The victoria-metrics-operator Cozystack already runs (appVersion `v0.68.4`, `packages/system/victoria-metrics-operator`) ships the VictoriaTraces CRDs `VTCluster` and `VTSingle` (`packages/system/victoria-metrics-operator/charts/victoria-metrics-operator/crd.yaml`, CRDs `vtclusters.operator.victoriametrics.com` and `vtsingles.operator.victoriametrics.com`). `VTCluster` decomposes into VTInsert/VTStorage/VTSelect components — conceptually analogous to `VLCluster`'s insert/storage/select — but note the exact spec keys differ: `VLCluster` uses `vlinsert`/`vlselect`/`vlstorage`, whereas `VTCluster` **drops the prefix** and uses `spec.insert`/`spec.select`/`spec.storage` (verified against the CRD's printer columns `.spec.insert.replicaCount` etc.). The rendered workloads/services are *expected* to carry the `vt` prefix (`vtinsert-*`/`vtselect-*`/`vtstorage-*`), same as `VLCluster` yields `vlinsert-*` — this prefix is load-bearing for the Stage A `otel-traces:4317` target, so it stays a **confirm-before-implementation** item (see Open questions) rather than an asserted fact. So the template is *shaped* like `vlogs.yaml` but the sub-spec keys are not a verbatim copy — this is the one place the analogy misleads, and it is corrected throughout below. ### The problem @@ -63,6 +63,7 @@ Add a `tracingStorages` list to the monitoring values, parallel to `metricsStora ```yaml tracingStorages: - name: generic + mode: cluster # "cluster" -> VTCluster (default), "single" -> VTSingle (edge/dev/small) retentionPeriod: "14d" # age-based retention; default 14 days per the requirement retentionDiskUsageBytes: "" # optional disk-based cap, byte quantity (e.g. "50GB"); see note below storageSize: 10Gi # PVC size for vtstorage @@ -70,13 +71,14 @@ tracingStorages: replicaCount: 2 # component scaling, NOT data replication (see durability note) ``` -Render one `VTCluster` per entry in a new `templates/vtraces/vtraces.yaml`, shaped like `templates/vlogs/vlogs.yaml` (but with `VTCluster`'s prefix-less sub-spec keys): `managedMetadata` labels for application ownership, configurable replica counts on `spec.insert`/`spec.select`/`spec.storage`, `spec.storage.retentionPeriod` from the entry, the storage-PVC claim-template label for the release-scoped post-delete cleanup hook, and the `cozystack/cozystack#3181` guard (see the absent-vs-empty contract below). +Render one backend CR per entry in a new `templates/vtraces/vtraces.yaml`, shaped like `templates/vlogs/vlogs.yaml` (but with the VictoriaTraces prefix-less sub-spec keys). The per-entry `mode` selects the CR kind: `cluster` (default) renders a `VTCluster` — the multi-component variant used below and the same choice `metricsStorages`/`logsStorages` make with `VMCluster`/`VLCluster` in every stack today; `single` renders a single-binary `VTSingle` for edge/dev/small clusters where the cluster footprint is unwarranted (its sub-spec is flatter — no separate insert/select/storage components — so `replicaCount` and the storage/retention fields apply to the one workload). Both carry: `managedMetadata` labels for application ownership, configurable replica counts, `retentionPeriod` from the entry, the storage-PVC claim-template label for the release-scoped post-delete cleanup hook, and the `cozystack/cozystack#3181` guard (see the absent-vs-empty contract below). **Absent vs empty contract** (the two must not be conflated): an **absent/unset** `tracingStorages` key means "tracing disabled", and the template renders nothing and succeeds (so tracing stays opt-in and a cluster that never wanted traces is never broken). An **explicitly empty list** (`tracingStorages: []`) is a misconfiguration — a consumer asked for tracing but declared no backend — and **fails the render**, exactly like the logs `#3181` guard. Both cases must be covered by tests. **Durability note:** `replicaCount` scales the number of `vtstorage` pods for throughput/availability of the *component*, but VictoriaTraces cluster mode does **not** replicate span data between storage nodes — losing a storage node can make some queries return partial results. This is the same durability model as `VLCluster`, and it is *not* HA data replication. Real cross-node durability requires either replication through Collectors into two independent VictoriaTraces backends or an equivalent design; this is called out in Open questions rather than solved here. ```yaml +{{- /* sketch shows mode: cluster; mode: single renders kind: VTSingle with the flatter single-binary spec */}} {{- range .Values.tracingStorages }} --- apiVersion: operator.victoriametrics.com/v1 @@ -122,7 +124,7 @@ The monitoring HelmRelease must gate readiness on the new `VTCluster` exactly as **OTLP service contract** (the exact ports/path matter, and a bare `ExternalName` cannot carry them): an `ExternalName` only aliases DNS; it does not remap ports or translate the OTLP/HTTP path. `vtinsert` defaults to OTLP/HTTP on port `10481` at path `/insert/opentelemetry/v1/traces`, and to OTLP/gRPC only when `-otlpGRPCListenAddr` is set (conventionally `4317`). Because the collector's OTLP/HTTP path (`/v1/traces`) differs from `vtinsert`'s, **OTLP/gRPC is the chosen stable app contract**: gRPC has no path, so the same `…:4317` Service works whether it fronts `vtinsert` (Stage A) or the collector (Stage B). The redirect is therefore an explicit ports-carrying Service (`ClusterIP` with an `externalName`-style upstream, or a small proxy), never plain DNS aliasing. OTLP/HTTP users must standardize on the collector from the start, since its ingest path is not interchangeable with `vtinsert`'s. -**Tenant routing needs headers** (the load-bearing multi-tenancy point): VictoriaTraces attributes a tenant from the `AccountID`/`ProjectID` request headers; with neither set, everything lands in the default tenant `0:0`. Neither an `ExternalName` redirect nor a collector `resource` processor sets those headers. So on a shared central backend the write path must inject `AccountID`/`ProjectID` derived from authenticated workload identity — from the app's OTLP exporter, a trusted per-tenant proxy, or (Stage B) the collector's `headers_setter` extension (see below) — in addition to stamping the `tenant` resource attribute for query-time labelling. Isolated per-tenant stacks (`packages/extra/monitoring`) sidestep this by not sharing a backend. +**Tenant routing needs headers** (the load-bearing multi-tenancy point): VictoriaTraces attributes a tenant from the `AccountID`/`ProjectID` request headers; with neither set, everything lands in the default tenant `0:0`. Note the current Cozystack convention is precisely account `0` everywhere plus a `tenant` external label (see Context) — the same one metrics/logs use — so per-tenant `AccountID`/`ProjectID` is *not* today's convention but the option a **shared** trace backend would adopt to separate tenants by account. Neither an `ExternalName` redirect nor a collector `resource` processor sets those headers. So on a shared central backend the write path must inject `AccountID`/`ProjectID` derived from authenticated workload identity — from the app's OTLP exporter, a trusted per-tenant proxy, or (Stage B) the collector's `headers_setter` extension (see below) — in addition to stamping the `tenant` resource attribute for query-time labelling. Isolated per-tenant stacks (`packages/extra/monitoring`), the default isolation model, sidestep this by not sharing a backend. **Stage B — toggleable OpenTelemetry Collector gateway.** Add an OpenTelemetry Collector (gateway **Deployment** in `cozy-monitoring`, following the `packages/system/monitoring-agents` pattern) in front of `vtinsert` for the capabilities direct ingest can't provide: central sampling, rate-limiting, and per-`tenant` resource attribution. It is the "OpenTelemetry Collector/Agent via an option" the client explicitly requested, and it is opt-in. @@ -136,13 +138,13 @@ Because the app-facing contract is the fixed `otel-traces…:4317` gRPC Service, ### 3. Grafana datasource and correlation -Add a `GrafanaDatasource` CR per `tracingStorages` entry in `templates/vtraces/grafana-datasource.yaml`, mirroring the logs datasource template and attaching through `instanceSelector: { matchLabels: { dashboards: grafana } }`. VictoriaTraces exposes a Jaeger-compatible query API, so the datasource is `type: jaeger` (or the dedicated VictoriaTraces datasource plugin, allow-listed like `victoriametrics-logs-datasource` is today) pointed at the `vtselect` service. Configure: +Add a `GrafanaDatasource` CR per `tracingStorages` entry in `templates/vtraces/grafana-datasource.yaml`, mirroring the logs datasource template and attaching through `instanceSelector: { matchLabels: { dashboards: grafana } }`. VictoriaTraces exposes a Jaeger-compatible query API under the `/select/jaeger` prefix on `vtselect`, so the datasource is `type: jaeger` (or the dedicated VictoriaTraces datasource plugin, allow-listed like `victoriametrics-logs-datasource` is today) with a same-namespace URL — `http://vtselect-{{ .name }}.{{ $.Release.Namespace }}.svc:10471/select/jaeger` (cluster; `:10428` for `VTSingle`) — exactly as the logs datasource points at its in-namespace `vlselect`. Configure: - **Trace → logs**: link to the VictoriaLogs datasource keyed on `trace_id`. - **Trace → metrics**: link to the VictoriaMetrics datasource for RED-style span metrics. Grafana's trace-to-metrics only *links* to pre-existing metrics — it does not generate them. So RED/span metrics need a source: the Stage-B collector's `spanmetrics` connector (recommended, produces RED metrics from spans and exports them to VictoriaMetrics), or native app instrumentation. Under Stage A (direct ingest, no collector) there are no span metrics — trace↔metrics correlation is therefore a Stage-B capability, and exemplars linking metrics→trace likewise depend on the metric producer emitting `trace_id` exemplars. - **Logs/metrics → trace**: derived fields on the existing datasources so a `trace_id` in a log or exemplar opens the trace. -**Read-side tenant isolation** (the gap most likely to bite in central-backend mode): the write path attributes tenants via `AccountID`/`ProjectID`, but the *read* path must also scope a tenant's Grafana traces datasource to its own `AccountID`/`ProjectID` so tenant A cannot query tenant B's spans through the shared `vtselect`. This mirrors how logs/metrics rely on per-tenant labelling and/or per-tenant stacks; for traces on a shared backend it means the per-tenant `GrafanaDatasource` must carry the tenant's account/project scoping (or the tenant runs an isolated stack). This is called out explicitly in Open questions. +**Read-side tenant isolation** works exactly as it does for logs and metrics today — **structurally, not by a query-time tenant filter**. Cozystack does not scope reads with `AccountID`/`ProjectID` anywhere: every monitoring backend uses account `0`, tenants that need isolation run their own self-contained stack in their own namespace (`packages/extra/monitoring`, wired by `packages/apps/tenant/templates/monitoring.yaml`), and the Grafana datasource URL is pinned to that namespace's own select service, fenced by NetworkPolicy. Traces inherit this model unchanged: the isolated per-tenant trace stack's datasource points at its own in-namespace `vtselect`, so tenant A physically cannot reach tenant B's spans. This is important because `vtselect` performs **no per-tenant authorization** and trusts `AccountID`/`ProjectID` verbatim (upstream: "use vmauth for per-tenant authorization") — so a datasource header pin is a routing selector, never a security boundary. On a *shared* central backend, traces get the same weak, label-only cross-tenant read property that shared metrics/logs already have; making that a real boundary would require an authenticating proxy (vmauth) that forces the headers from identity and strips client-supplied ones — deferred as future work, not part of the MVP (see Resolved / Open questions). ### 4. Per-application opt-in @@ -194,7 +196,7 @@ The change is purely additive and opt-in. Existing clusters see no behavioural c - **New tenant-supplied input**: the OTLP endpoint accepts spans from tenant workloads. In Stage A the shared `vtinsert` backend is the exposed surface (relying on VictoriaTraces' own limits and the per-tenant isolation of isolated stacks); Stage B's collector becomes the explicit trust boundary — enforcing per-tenant attribution, rate-limits, and sampling so a noisy or hostile tenant cannot exhaust the shared backend. This hardening is the main reason to promote Stage B on a shared central backend. - **Tenant attribution must be trusted**: `AccountID`/`ProjectID` (and the `tenant` attribute) must be injected from authenticated workload identity, never accepted verbatim from a tenant that could spoof another's IDs. On the shared backend this injection belongs at a boundary the tenant cannot bypass — the per-tenant proxy or the collector — not purely in tenant-controlled app config. -- **Isolation (write and read)**: write-side attribution alone is not isolation. The **read** path must scope each tenant's Grafana traces datasource to its own `AccountID`/`ProjectID` so a shared `vtselect` cannot leak spans across tenants; central-backend mode keeps the existing tenant-labelling model, isolated mode (`packages/extra/monitoring`) keeps traces inside the tenant. +- **Isolation (write and read)**: write-side attribution alone is not isolation, and `vtselect` performs no per-tenant authorization — it trusts the `AccountID`/`ProjectID` read header verbatim, so pinning that header on a per-tenant Grafana datasource is a routing selector, not a security boundary. Read isolation is therefore delivered the same way logs/metrics deliver it today: **either** (a) an isolated per-tenant stack (`packages/extra/monitoring`) whose datasource can only reach its own in-namespace `vtselect`, fenced by NetworkPolicy — the recommended default; **or** (b) an authenticating proxy (vmauth) in front of a shared `vtselect` that derives `AccountID`/`ProjectID` from authenticated identity and strips any client-supplied headers. A per-tenant datasource header pin is safe only in combination with (a) or (b), because the endpoint itself remains reachable and unauthenticated. - **Transport**: OTLP endpoints should be TLS-terminated; align with the unified TLS/PKI model (`design-proposals/unified-tls-pki`) rather than minting bespoke certs. - **RBAC**: new `VTCluster`/`GrafanaDatasource`/collector resources need the same narrowly-scoped RBAC the metrics/logs equivalents already have. No new secret classes are introduced beyond the OTLP endpoint credentials, if any. @@ -204,7 +206,7 @@ The change is purely additive and opt-in. Existing clusters see no behavioural c - OTLP endpoint unreachable from an app (backend or collector down) → the app's exporter drops/retries per OTLP defaults; the app itself does not crash and serving is unaffected. - Storage full — **age vs disk retention are independent**. `retentionPeriod` only prunes by age; it does *not* bound disk. Without a disk cap a full PVC blocks ingest *before* old traces are evicted. Set `spec.storage.retentionMaxDiskSpaceUsageBytes` (the disk-cap field the `VTCluster` CRD actually exposes — verified in the pinned operator; a percent form is *not* a CR field, only an upstream flag) via the byte-quantity `retentionDiskUsageBytes` values field, and add a PVC/disk-usage capacity alert. Covered by a PVC-exhaustion test. - `tracing.enabled: false` → no sidecar, no env, no CR: zero overhead. -- App emits OTLP but no backend deployed → the ExternalName resolves to nothing and the exporter drops/retries harmlessly; the app toggle is documented to require a `tracingStorages` backend. +- App emits OTLP but no backend deployed → the `otel-traces` Service has no backing endpoints, so the exporter's connection fails and it drops/retries harmlessly; the app toggle is documented to require a `tracingStorages` backend. - App sets no `AccountID`/`ProjectID` → spans land in the default tenant `0:0` (visible to whoever can read that tenant). The write-boundary injection above must prevent this on a shared backend. ## Testing @@ -224,15 +226,13 @@ The change is purely additive and opt-in. Existing clusters see no behavioural c ## Open questions - **VictoriaTraces maturity**: the CRDs ship in the operator, but is VictoriaTraces production-ready at the version Cozystack pins? If not, Grafana Tempo is the drop-in fallback (see Alternatives) — the collector and per-app surfaces are backend-agnostic, so only the backend template and datasource type change. -- **`VTCluster` vs `VTSingle`**: lean toward supporting **both** (as metrics/logs do) — `VTSingle` for edge/dev/small clusters to cut overhead, `VTCluster` for production. Which is the default per stack? - **Confirm-before-implementation (upstream VictoriaTraces surface)**: the `VTCluster` spec keys (`insert`/`select`/`storage`) and the disk-cap field (`retentionMaxDiskSpaceUsageBytes`) are verified against the pinned operator CRD. Still taken on upstream faith and to be confirmed against the pinned VictoriaTraces version before building: the Grafana datasource type (`type: jaeger` against `vtselect`, or the dedicated VictoriaTraces plugin — the whole correlation UX depends on which lands); the OTLP wire details (`vtinsert` HTTP `:10481` + `/insert/opentelemetry/v1/traces`, gRPC via `-otlpGRPCListenAddr`); and the exact rendered service-name prefix (`vtinsert-*`/`vtselect-*` vs `insert-*`). -- **Read-side tenant isolation on a shared backend**: how exactly does the per-tenant Grafana datasource scope a tenant to its own `AccountID`/`ProjectID` on `vtselect`, so tenant A cannot read tenant B's spans? (Isolated per-tenant stacks avoid the question; the shared central backend does not.) - **Sampling default**: head sampling at the app (`probabilistic_sampler`, scales freely) vs tail sampling at the collector (needs trace affinity — single replica or `loadbalancingexporter`)? Only relevant once Stage B lands. - **Durability**: is single-instance `vtstorage` acceptable, or does the platform need cross-node span durability (collector replication into two independent backends)? - **External OTLP exposure**: should tenants be able to push spans from outside the cluster, and if so through which ingress/Gateway path? - **Stage-B trigger**: what concrete signal (backend load, abuse, the tenant-header requirement) promotes the collector gateway from optional to default? Note that shared-backend multi-tenancy effectively *needs* the collector (or a trusted proxy) to inject `AccountID`/`ProjectID`, so Stage B may be mandatory for a shared central backend rather than purely optional. -Resolved during design (recorded here so they are not re-litigated): ingest is **staged A→B** — direct-to-`vtinsert` first, collector gateway as an opt-in second — because `vtinsert` is OTLP-native and a **port-explicit OTLP/gRPC `:4317` Service** (not a bare `ExternalName`) keeps the app-facing endpoint stable across the switch. The collector is a gateway **Deployment**, not a DaemonSet, because OTLP is push-based (apps are the agents). Tail sampling, if used, runs single-replica or behind a `loadbalancingexporter` (`routing_key: traceID`) to stay trace-affine. The `tracingStorages` contract is absent→disabled-and-render-ok, explicit-empty→fail. Tracing opt-in lives in app `values.yaml`, not `WorkloadMonitor` (operational-only). +Resolved during design (recorded here so they are not re-litigated): ingest is **staged A→B** — direct-to-`vtinsert` first, collector gateway as an opt-in second — because `vtinsert` is OTLP-native and a **port-explicit OTLP/gRPC `:4317` Service** (not a bare `ExternalName`) keeps the app-facing endpoint stable across the switch. The collector is a gateway **Deployment**, not a DaemonSet, because OTLP is push-based (apps are the agents). Tail sampling, if used, runs single-replica or behind a `loadbalancingexporter` (`routing_key: traceID`) to stay trace-affine. The `tracingStorages` contract is absent→disabled-and-render-ok, explicit-empty→fail. Tracing opt-in lives in app `values.yaml`, not `WorkloadMonitor` (operational-only). Both `VTCluster` and `VTSingle` are supported via a per-entry `mode` field (default `cluster`, matching how `metricsStorages`/`logsStorages` render the cluster variant in every stack today; `single` for edge/dev/small). Read-side tenant isolation is **structural**, exactly as for logs/metrics: account `0` everywhere, isolation by an isolated per-tenant stack (`packages/extra/monitoring`) whose datasource reaches only its own in-namespace `vtselect`, fenced by NetworkPolicy — not a query-time tenant filter, since `vtselect` has no per-tenant authorization and trusts the `AccountID`/`ProjectID` header verbatim. A shared central backend would need vmauth (forcing headers from authenticated identity, stripping client-supplied ones) to become a real read boundary; that is deferred future work, not the MVP. ## Alternatives considered From a538e0b59bd6139947c39f2f5c3fb6ffbc97e517 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Thu, 23 Jul 2026 16:49:32 +0300 Subject: [PATCH 10/10] Address Mattia + IvanHunters review: disk-bounded default, per-tier replicas, storageSize rationale, PSS MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Mattia #1: default is always disk-bounded — retentionMaxDiskSpaceUsageBytes derived from storageSize (~85%) when retentionDiskUsageBytes is unset, so the out-of-the-box config can't silently fill the PVC. Update Failure + Testing. - Mattia #2: replicaCount documented as a uniform shortcut with optional per-tier insert/select/storage overrides (as VMCluster), storage tier durability-adjacent. - IvanHunters: inline rationale for the storageSize key name (avoids spec.storage.storage stutter with the VTCluster CRD's nested storage object). - IvanHunters: add Pod Security Standards posture for the new Collector Deployment and per-app OTLP sidecars (restricted: non-root, drop ALL, seccomp RuntimeDefault, no privilege escalation). - Bump revised-date to 2026-07-23. Signed-off-by: Alexey Artamonov --- .../distributed-tracing/README.md | 30 +++++++++++-------- 1 file changed, 18 insertions(+), 12 deletions(-) diff --git a/design-proposals/distributed-tracing/README.md b/design-proposals/distributed-tracing/README.md index 996eac8..2df86b8 100644 --- a/design-proposals/distributed-tracing/README.md +++ b/design-proposals/distributed-tracing/README.md @@ -2,7 +2,7 @@ - **Title:** `Distributed tracing for managed applications via OTLP and VictoriaTraces` - **Author(s):** `@scooby87` -- **Date:** `2026-07-16` (revised 2026-07-20) +- **Date:** `2026-07-16` (revised 2026-07-23) - **Status:** Review ## Overview @@ -65,12 +65,17 @@ tracingStorages: - name: generic mode: cluster # "cluster" -> VTCluster (default), "single" -> VTSingle (edge/dev/small) retentionPeriod: "14d" # age-based retention; default 14 days per the requirement - retentionDiskUsageBytes: "" # optional disk-based cap, byte quantity (e.g. "50GB"); see note below - storageSize: 10Gi # PVC size for vtstorage + retentionDiskUsageBytes: "" # disk-based cap, byte quantity (e.g. "50GB"); when empty, derived from storageSize (~85%) so the default is always disk-bounded (see note) + storageSize: 10Gi # PVC size for vtstorage (named storageSize, not storage — see naming note) storageClassName: "" - replicaCount: 2 # component scaling, NOT data replication (see durability note) + replicaCount: 2 # uniform shortcut for all tiers; NOT data replication (see durability note) + # insert: { replicaCount: 2 } # optional per-tier override (defaults to replicaCount) + # select: { replicaCount: 2 } # optional per-tier override (defaults to replicaCount) + # storage: { replicaCount: 2 } # optional per-tier override; durability-adjacent, still NOT replication ``` +The per-entry `replicaCount` is a **uniform shortcut**: it seeds all three cluster tiers so a simple config stays one line. Because the tiers are rarely sized identically in practice — `insert` scales with ingest, `select` with query load, `storage` with retention/throughput, and `storage.replicaCount` is durability-adjacent rather than a query-scaling knob — each tier accepts an optional explicit override (`insert`/`select`/`storage` `replicaCount`), mirroring how `VMCluster` exposes per-component counts; an unset tier falls back to the shortcut. The PVC-size key is named `storageSize` rather than the sibling `storage` (used by `metricsStorages`/`logsStorages`) on purpose: the `VTCluster` CRD already nests a second `storage` object under `spec.storage` (`spec.storage.storage.volumeClaimTemplate`), so reusing `storage` as the values key would read as `spec.storage.storage` stutter — the rename keeps the values surface unambiguous. + Render one backend CR per entry in a new `templates/vtraces/vtraces.yaml`, shaped like `templates/vlogs/vlogs.yaml` (but with the VictoriaTraces prefix-less sub-spec keys). The per-entry `mode` selects the CR kind: `cluster` (default) renders a `VTCluster` — the multi-component variant used below and the same choice `metricsStorages`/`logsStorages` make with `VMCluster`/`VLCluster` in every stack today; `single` renders a single-binary `VTSingle` for edge/dev/small clusters where the cluster footprint is unwarranted (its sub-spec is flatter — no separate insert/select/storage components — so `replicaCount` and the storage/retention fields apply to the one workload). Both carry: `managedMetadata` labels for application ownership, configurable replica counts, `retentionPeriod` from the entry, the storage-PVC claim-template label for the release-scoped post-delete cleanup hook, and the `cozystack/cozystack#3181` guard (see the absent-vs-empty contract below). **Absent vs empty contract** (the two must not be conflated): an **absent/unset** `tracingStorages` key means "tracing disabled", and the template renders nothing and succeeds (so tracing stays opt-in and a cluster that never wanted traces is never broken). An **explicitly empty list** (`tracingStorages: []`) is a misconfiguration — a consumer asked for tracing but declared no backend — and **fails the render**, exactly like the logs `#3181` guard. Both cases must be covered by tests. @@ -91,14 +96,14 @@ spec: apps.cozystack.io/application.group: apps.cozystack.io apps.cozystack.io/application.kind: Monitoring apps.cozystack.io/application.name: {{ $.Release.Name }} - insert: { replicaCount: {{ .replicaCount | default 2 }} } - select: { replicaCount: {{ .replicaCount | default 2 }} } + {{- /* per-tier replicaCount falls back to the uniform .replicaCount shortcut (fallback plumbing elided in this sketch) */}} + insert: { replicaCount: {{ .insert.replicaCount | default .replicaCount | default 2 }} } + select: { replicaCount: {{ .select.replicaCount | default .replicaCount | default 2 }} } storage: retentionPeriod: {{ .retentionPeriod | quote }} - {{- with .retentionDiskUsageBytes }} - retentionMaxDiskSpaceUsageBytes: {{ . | quote }} - {{- end }} - replicaCount: {{ .replicaCount | default 2 }} + {{- /* always disk-bounded: explicit bytes, else ~85% of storageSize so the default cannot silently fill the PVC (exact byte math elided) */}} + retentionMaxDiskSpaceUsageBytes: {{ .retentionDiskUsageBytes | default (include "vtraces.defaultDiskCap" .) | quote }} + replicaCount: {{ .storage.replicaCount | default .replicaCount | default 2 }} storage: volumeClaimTemplate: metadata: @@ -199,19 +204,20 @@ The change is purely additive and opt-in. Existing clusters see no behavioural c - **Isolation (write and read)**: write-side attribution alone is not isolation, and `vtselect` performs no per-tenant authorization — it trusts the `AccountID`/`ProjectID` read header verbatim, so pinning that header on a per-tenant Grafana datasource is a routing selector, not a security boundary. Read isolation is therefore delivered the same way logs/metrics deliver it today: **either** (a) an isolated per-tenant stack (`packages/extra/monitoring`) whose datasource can only reach its own in-namespace `vtselect`, fenced by NetworkPolicy — the recommended default; **or** (b) an authenticating proxy (vmauth) in front of a shared `vtselect` that derives `AccountID`/`ProjectID` from authenticated identity and strips any client-supplied headers. A per-tenant datasource header pin is safe only in combination with (a) or (b), because the endpoint itself remains reachable and unauthenticated. - **Transport**: OTLP endpoints should be TLS-terminated; align with the unified TLS/PKI model (`design-proposals/unified-tls-pki`) rather than minting bespoke certs. - **RBAC**: new `VTCluster`/`GrafanaDatasource`/collector resources need the same narrowly-scoped RBAC the metrics/logs equivalents already have. No new secret classes are introduced beyond the OTLP endpoint credentials, if any. +- **Pod Security Standards**: the new OTLP Collector Deployment and the per-app OTLP sidecars/agents (Kafka, RabbitMQ, MariaDB, Redis, Postgres) run inside `tenant-*` namespaces, which enforce PSS **restricted**. They must ship a compliant pod-security posture out of the box — `runAsNonRoot: true`, `allowPrivilegeEscalation: false`, `capabilities.drop: ["ALL"]`, and `seccompProfile.type: RuntimeDefault` — like every other new workload class in this stack; no privileged or host-namespace access is required for OTLP ingest. ## Failure and edge cases - `tracingStorages` **absent/unset** → tracing disabled, template renders nothing and succeeds. `tracingStorages: []` **explicitly empty** → loud render failure (mirrors `cozystack/cozystack#3181`), never a silent span black hole. Both paths are asserted in tests. - OTLP endpoint unreachable from an app (backend or collector down) → the app's exporter drops/retries per OTLP defaults; the app itself does not crash and serving is unaffected. -- Storage full — **age vs disk retention are independent**. `retentionPeriod` only prunes by age; it does *not* bound disk. Without a disk cap a full PVC blocks ingest *before* old traces are evicted. Set `spec.storage.retentionMaxDiskSpaceUsageBytes` (the disk-cap field the `VTCluster` CRD actually exposes — verified in the pinned operator; a percent form is *not* a CR field, only an upstream flag) via the byte-quantity `retentionDiskUsageBytes` values field, and add a PVC/disk-usage capacity alert. Covered by a PVC-exhaustion test. +- Storage full — **age vs disk retention are independent**. `retentionPeriod` only prunes by age; it does *not* bound disk, so an age-only config lets a full PVC block ingest *before* old traces are evicted. The default is therefore **always disk-bounded**: `spec.storage.retentionMaxDiskSpaceUsageBytes` (the disk-cap field the `VTCluster` CRD actually exposes — verified in the pinned operator; a percent form is *not* a CR field, only an upstream flag) is set from the byte-quantity `retentionDiskUsageBytes` values field when given, and otherwise **derived from `storageSize` (~85%)** so no entry ever ships without a disk bound — matching the "uncrashable default, don't just document the knife" spirit of the `#3181` guard. A PVC/disk-usage capacity alert is added on top, and the behaviour is covered by a PVC-exhaustion test. - `tracing.enabled: false` → no sidecar, no env, no CR: zero overhead. - App emits OTLP but no backend deployed → the `otel-traces` Service has no backing endpoints, so the exporter's connection fails and it drops/retries harmlessly; the app toggle is documented to require a `tracingStorages` backend. - App sets no `AccountID`/`ProjectID` → spans land in the default tenant `0:0` (visible to whoever can read that tenant). The write-boundary injection above must prevent this on a shared backend. ## Testing -- **Unit/lint**: `helm template` + `helm lint` for the new `vtraces` templates and each app's `tracing` block; assert `VTCluster`, the collector, and the `GrafanaDatasource` render only when configured; assert the **absent-vs-empty** contract (absent `tracingStorages` → renders nothing and succeeds; `tracingStorages: []` → render fails); assert `retentionDiskUsageBytes` maps to `spec.storage.retentionMaxDiskSpaceUsageBytes` when set. +- **Unit/lint**: `helm template` + `helm lint` for the new `vtraces` templates and each app's `tracing` block; assert `VTCluster`, the collector, and the `GrafanaDatasource` render only when configured; assert the **absent-vs-empty** contract (absent `tracingStorages` → renders nothing and succeeds; `tracingStorages: []` → render fails); assert `retentionDiskUsageBytes` maps to `spec.storage.retentionMaxDiskSpaceUsageBytes` when set, and that when it is unset the field is still rendered, derived from `storageSize` (~85%), so no rendered `VTCluster`/`VTSingle` is ever disk-unbounded. - **e2e** (Chainsaw, per `docs/agents/e2e-testing.md`): deploy the monitoring stack with a `tracingStorages` entry, deploy one app (start with a native-OTLP engine, e.g. ClickHouse) with `tracing.enabled: true`, generate activity, then assert a trace is queryable via the `vtselect` query API and visible in Grafana; assert a second tenant cannot read the first tenant's spans (read-side isolation); a PVC-exhaustion case asserts ingest degrades safely with a disk cap set. - **Manual**: verify trace→logs and (once the collector/spanmetrics land) trace→metrics pivots in Grafana.