Add CLI tunnel and auth commands

[drewr]

Summary

This PR ships the CLI client for Datum Connect tunneling — the headless equivalent of the desktop UI. It lets users authenticate, manage projects, and expose local services to public hostnames without launching the GUI.

Building

Rust tooling only (no Nix required):

cargo run -p datum-connect -- --help

Or with Nix:

nix run .#cli -- --help

Commands

auth

datum-connect auth login       # OAuth via browser; prompts to select a project after login
datum-connect auth logout
datum-connect auth status      # Shows authenticated user and active org/project
datum-connect auth list
datum-connect auth switch      # Logs out and re-authenticates; prompts to select a project

projects

datum-connect projects list    # Lists all orgs and projects; marks the active one with *
datum-connect projects switch  # Interactive prompt to change the active project

tunnel

datum-connect tunnel listen --endpoint 127.0.0.1:8080
datum-connect tunnel listen --endpoint 127.0.0.1:8080 --label my-tunnel
datum-connect tunnel listen --endpoint 127.0.0.1:8080 --project <project-id>
datum-connect tunnel list
datum-connect tunnel update --id <id> --label new-name
datum-connect tunnel delete --id <id>

tunnel listen runs in the foreground. It creates or reuses a tunnel for the given endpoint, starts the heartbeat agent so the gateway has routing info, enables the tunnel, and polls until it is accepted and programmed before printing the public hostname. Ctrl+C disables the tunnel and exits.

The --project flag overrides the active project for a single invocation without changing the stored selection.

Project selection

The active project is stored in config.yml (default: ~/.local/share/Datum/config.yml, overridable via $DATUM_CONNECT_REPO). It is set interactively after auth login or auth switch, or explicitly with projects switch.

Example session

$ cargo run -p datum-connect -- auth login
# browser opens for OAuth
Logged in as Jane Smith (jane@example.com)

Select a project:
  [1] Acme Corp / production
  [2] Acme Corp / staging
Enter number [1-2]: 2
Selected project: Acme Corp / staging

$ cargo run -p datum-connect -- tunnel listen --endpoint 127.0.0.1:3000
Created tunnel:
  id: httpp-abc123
  label: f3a9c2e1b047

Your endpoint ID: 30a9ddf5...
Setting up tunnel...
Tunnel ready after 8 sec: https://f3a9c2e1b047.tunnels.datum.net
Press Ctrl+C to stop...

Bug fixes (found during testing)

• Tunnels created from CLI never route traffic: CLI was missing the HeartbeatAgent that continuously patches status.connectionDetails on the connector. Without it the gateway has no routing info. Fixed: tunnel listen now starts the heartbeat and registers the project before enabling the tunnel.
• Re-running tunnel listen on an existing endpoint always prompted for update: Random label was generated before checking for an existing tunnel, so it always differed. Fixed: label generation moved into the create-new path; existing tunnels reuse their stored label unless --label is explicitly given.
• Tunnel delete silently no-ops when connector is missing: delete_project returned early if no connector was found, skipping deletion of HTTPProxy/ConnectorAdvertisement/TrafficProtectionPolicy. Fixed: connector lookup is only needed for post-deletion cleanup and no longer gates resource deletion.
• Auto-generated label used tunnel-<u16> format: Collided visually with resource ID format. Switched to 12 hex chars of random entropy (e.g. a3f9c2e1b047).

Test plan

• cargo run -p datum-connect -- auth login completes OAuth and prompts for project selection
• projects list shows all orgs/projects with active one marked
• projects switch persists new selection to config.yml
• tunnel listen --endpoint 127.0.0.1:<port> creates tunnel, prints hostname, disables on Ctrl+C
• Re-running tunnel listen on the same endpoint reuses the existing tunnel without prompting
• tunnel listen --project <id> uses the specified project
• tunnel list shows tunnels in the active project
• tunnel delete removes a tunnel cleanly

[zachsmith1]

Do we want a separate cli for tunnels or do we want to bake in functionality into datumctl?

[drewr]

Yeah, it's why this is a draft. I needed the functionality and didn't want to commit one way or the other yet. I explored doing it in datumctl and it would involve either replicating the Iroh sidecar in go or making the project hybrid with a rust component.

This method uses all the same machinery as the GUI which felt like a better first pass.

[zachsmith1]

Ya the challenge is the core stuff we need is in rust so we'll need some magic to make the UX good

[scotwells]

How does this interact with the GUI based application? Would auth be shared?

Since the GUI is locked to a specific project (because connectors are project-scoped resources), switching the authenticated user could break existing tunnels without the user knowing and it doesn't seem like we warn the user.

[drewr]

It's all shared. I'll show what it looks like when Rust is done compiling...

[drewr]

Here's a short demo of where I've gotten with this:

headless-tunnels-demo.mp4

[bmertens-datum]

@drewr Nice demo.

[zachsmith1]

@drewr this is slick. lm planning on splitting off the app repo from the gateway repo and we should consider where we'd want this cli to live. last piece there would be a small enhancement around how we could inject this rust binary into datumctl (if we want to)

[richardhenwood]

I've had a moment to try this - and following your excellent demo video, and typing datum-connect -- tunnel listen --endpoint localhost:8080 I got a 'connector' appearing in the Datum cloud UI. This is a really powerful way to think about connections for me - so I'm very excited to play around :)

FYI: this is on my Fedora 43 workstation.

[drewr]

Great feedback @richardhenwood, thanks!

[drewr]

@zachsmith1 wrote:

where we'd want this cli to live

I think if we factor out the local process to a standalone rust utility like you're proposing it makes more sense for this to live in datumctl. I originally went that direction but didn't want to either rewrite the iroh integration in go or repackage this in an awkward way.

[drewr]

I've had some instability with this and had both gpt-5.4 and sonnet-4.6 chewing on it:

Found it. The UpstreamProxy authorizes incoming iroh connections by checking self.state.get().proxies, but the CLI tunnel listen flow never calls listen.set_proxy() to register the tunnel in local state. The gateway connects over iroh, the auth handler finds no matching proxy, returns Forbidden, and the gateway sees a connection reset.

Fix incoming.

[gianarb]

There is a lot to unwrap in my opinion here, a lot around product so I am not sure I have enough context to help here.

Something is an old discussion we had here datum-cloud/enhancements#582 if you look for the ecosystem chapter:

Now my attention turns to "do we want to keep consistency in the ecosystem?". Do we want for example to get Datum Desktop to look at that file as well? So a switch context in the CLI will switch context in desktop?

In practice what I was trying to highlight here is the mood kubernetes and other cli tool develop when you do that everything you run starts from a unique source of truth (for kubernetes it is the ~/.kube/config file. If we can agree on something similar it will be a lot easier to bring other CTL or applications into a consistent state.

It will feel a lot easier to push for a plugin ecosystem like the one kubectl and others developed where binaries starting with kubectl- gets called from the main ctl. In this case we can release a binary datumctl-connect that will be callable like datumctl connect.

But if we can not agree on some common practices, like authentication the outcome for a user will be pretty poor, in this case I feel like we should just "give up" and release different binaries working their own way.

I am not saying that we should have in place the ability to switch and persist in between accounts/instances because I know we do not know yet datum-cloud/enhancements#653 (comment) but maybe since we do not know we can just take what we have today as common denominator until we figure out what's next.

So the way I envision the evolution of this PR is a binary that serves only the business logic to manage tunnels and connections and demands authentication to the same login used by the datumctl (or the datumctl changes to turn to the same used here and from desktop)

This is what I am trying to push to but as I said product wise I am not sure I have enough context to push into a direction vs another.

[drewr]

Gateway hostname normalization — needs investigation

While debugging an "upstream connect error or disconnect/reset before headers" report, I found the root cause in commit e2d868d: the gateway sends CONNECT localhost:<port> in the iroh HTTP CONNECT request regardless of what address was stored in the ConnectorAdvertisement (in this case 127.0.0.1). The strict string comparison in tcp_proxy_exists failed, returning Forbidden, which the gateway surfaces as a connection reset.

The client-side fix normalizes localhost, 127.0.0.1, and ::1 to a canonical form at comparison time, which handles the mismatch. But the gateway behavior is worth examining:

Observable behavior (from /tmp/datum-2026-04-09T19:45:27+00:00.log, line 8132):

handle request req=HttpRequest { version: HTTP/1.1, headers: {}, uri: localhost:3300, method: CONNECT }

The ConnectorAdvertisement had address: "127.0.0.1", but the gateway sent CONNECT localhost:3300.

Questions for the gateway team:

• Is this intentional? Does the gateway always normalize 127.0.0.1 → localhost for loopback addresses?
• Should the gateway instead use the ConnectorAdvertisement address verbatim in the CONNECT target?
• If the gateway normalizes to localhost, should it normalize to 127.0.0.1 instead (since that's what users typically specify as the --endpoint)?

The client-side fix is defensive and correct either way, but if the gateway is doing unintended normalization, fixing it there would be cleaner and might surface other subtle issues.

[drewr]

Diagnosis notes from a debugging session

Over a single CLI session I hit three distinct failure modes that all presented the same way — "Tunnel ready" with the data plane silently dropping. Recording the chain here so the cause-effect is preserved alongside the fix commits.

1. Heartbeat wedge on deleted Lease

Symptom. Tunnel went silent after no visible errors; logs eventually showed a warn loop:

heartbeat: lease renew failed: ApiError: leases.coordination.k8s.io "datum-connect-jttwh" not found: NotFound

firing every 30s indefinitely.

Root cause. run_for_project cached the resolved lease_name once and the renew error arms unconditionally retained the cache. A server-side delete of the Lease (TTL cleanup, namespace reap, manual) put the loop in a state it couldn't recover from — the cache never cleared, so the loop never went back through the connector-probe / lease-resolve path at the top.

Fix. c901b01 classifies the kube error per arm: 404 → reset cache, 401 → force refresh + retain, anything else → retain. Three unit tests cover the decision.

2. Machine-wide iroh identity colliding across projects

Symptom. A freshly-created tunnel reported Tunnel ready cleanly, but curl returned Connection reset by peer at the TLS handshake. The Connector's status held:

Type: IrohDNSPublished     Status: False
Reason: DeferredToOwner
Message: iroh DNS record is owned by Connector /<other-project>/default/datum-connect-<other>

Root cause. Repo::listen_key() resolved to a single flat path (<repo>/listen_key) regardless of project. Running the CLI against multiple projects from the same machine registered the same iroh public key under multiple Connector objects; the iroh DNS controller (network-services-operator/internal/controller/iroh_dns_controller.go) explicitly treats that as a collision ("Multiple Connectors that share the same iroh keypair … collapse to one DNSRecordSet — the first to claim wins, and the loser surfaces a DeferredToOwner condition"). The losing project's Connector is Ready=True but unreachable.

Fix. 76987b7 scopes listen_key under <repo>/projects/<project_id>/ with a one-shot migration that moves the legacy flat key into the first project that requests it (so the user keeps continuity with one existing server-side Connector; the rest get fresh keys). connect_key and gateway_key stay flat — neither registers a per-project server-side identity.

The audit of network-services-operator confirmed per-project (per-Connector) identity is the intended design model, not just an acceptable workaround.

3. Setup readiness check too shallow

Symptom. Tunnels reported Tunnel ready after 2 sec even when the connector's IrohDNSPublished was DeferredToOwner — i.e., when they were guaranteed not to carry traffic.

Root cause. The setup loop only checked accepted && programmed && !hostnames.is_empty() on the HTTPProxy. The Connector's conditions weren't read at all, so collisions and stuck states never surfaced to the user.

Fix. 90a7ab3 exposes a typed TunnelProgress over six controller conditions (ProxyAccepted, CertificatesReady, ConnectorReady, IrohDnsPublished, ProxyProgrammed, ConnectorMetadataProgrammed) and streams each transition during setup. terminal_failure() matches specifically on IrohDNSPublished=False; DeferredToOwner and bails immediately rather than waiting forever.

4. Stale DNS owner re-emerging post-setup

Symptom. A tunnel that came up cleanly worked for ~9 minutes and then quietly went dark with no client-side signal. Ready=True, lease being heartbeated on schedule, no errors in CLI logs.

Root cause. A previously-deleted Connector's iroh DNS claim was never cleaned up server-side. When the iroh DNS controller re-reconciled, it flipped the live Connector's IrohDNSPublished from True back to False with DeferredToOwner, naming the dead Connector's UID as the owner. The DNS claim outliving its owning Connector looks like either a missed owner-reference cascade on Connector delete or a controller-side cache that isn't invalidated when the Connector goes away.

Fix (client-side, this PR). 6264818 adds a runtime watch alongside the login-state watch — poll get_active_progress every 10s after setup completes, surface terminal failures with the same format_terminal_failure message the setup path uses, and break out of the run loop so the disable/cleanup path runs. Tunnel-deleted-server-side also breaks out; transient query errors just warn and retry.

Server-side follow-up (out of scope for this PR). The iroh DNS controller leaking a claim past its owning Connector's lifetime is a real bug. Worth filing against the operator team — concrete instance from today's session was datum-connect-jttwh (UID 226a90b6-3cad-4242-9eff-c2c71a335545) showing up as Owner of a DNS record after the Connector itself returned 404.

Recovery playbook this PR enables

If a user's tunnel goes terminal mid-session, the CLI now:

1. Prints the operator's message naming the conflicting Connector,
2. Exits the run loop cleanly,
3. Disables the tunnel server-side on shutdown.

Operator action: delete the per-project listen_key (<repo>/projects/<project_id>/listen_key), then rerun. The new identity sidesteps the stale claim entirely.

[drewr]

Quota 403 diagnosis

While shipping d8f7c96 (CLI-side retry for the transient quota-check timeout), I traced the chain that produces this error:

ApiError: connectoradvertisements.networking.datumapis.com "tunnel-gchhg" is forbidden:
Your request took too long to be checked against your quota.
Please try again in a moment — if this keeps happening, contact support.: Forbidden (code: 403)

⚠ Best-effort diagnosis. Reconstructed from network-services-operator/config/quota/ resources, compute:api/v1alpha/instance_types.go reason codes, and engineering ops reviews. @drewr — please sanity-check against Milo internals (the actual webhook + quota backend code lives in a Milo repo I don't have local access to) before relying on this.

The chain

1. API server receives Create ConnectorAdvertisement.
2. Milo's quota admission webhook (quota.miloapis.com) intercepts.
3. Webhook reads the matching ClaimCreationPolicy (e.g. connector-claim-policy.yaml), constructs a ResourceClaim of amount: 1 against ResourceType: networking.datumapis.com/connectors.
4. Webhook calls a quota backend to check project remaining capacity.
5. If the backend round-trip exceeds the webhook's timeoutSeconds, admission fails closed and returns the 403 above.

The failure class is already named in datum-cloud code

From datum-cloud/compute:api/v1alpha/instance_types.go:

// InstanceQuotaGrantedReasonBackendUnavailable indicates quota enforcement
// is configured but the Milo quota backend is unreachable (network error,
// timeout, transient failure).
//
// InstanceQuotaGrantedReasonMisconfigured indicates the ResourceClaim was
// rejected by the Milo admission plugin (403/422): ResourceRegistration absent
// or the policy is malformed.

Our 403 is the user-facing surface of BackendUnavailable — the webhook itself is healthy, but the backend it consults is slow/unreachable.

Most likely root causes (in order of probability)

1. Webhook → quota backend round-trip latency. Admission webhooks have a hard cap (timeoutSeconds, max 30s). If the backend call routinely lands in the slow tail (GC pause on the backend pod, network blip, cold start), some fraction of admissions will time out. The user hitting this in a tight resume loop — same project, same resource type — is consistent with the backend being intermittently slow.
2. Expensive per-request counting. If the webhook computes "current usage" by listing/aggregating all ResourceClaims for the project each request, latency scales with project state. Caching the count between admissions would fix it.
3. Resource limits on the webhook or backend. CPU throttling or memory pressure on those pods drives request latency up unpredictably.
4. Storage-side latency on the quota service (etcd or whatever Milo's quota backend uses). The 5/18 ops review tracks 409s on quota.miloapis.com as elevated — suggests this surface has known stability concerns.

How to confirm

• Pull quota.miloapis.com webhook latency metrics — p50/p95/p99 over the period when this user hit it. If p99 is near the webhook's timeoutSeconds, that's the root cause.
• Check whether the webhook caches per-project usage or recomputes per admission.
• Inspect resource limits and throttling on the quota-webhook and quota-backend pods.
• Look at whether the webhook does any internal retry on transient backend failures before returning the user-visible 403.

Server-side fixes worth considering

(Not in this PR's scope — the actual implementation is in Milo — but flagging for whoever owns it.)

• Cache project quota usage in the webhook with a short TTL (1–5s); coalesce admissions of the same project so a burst doesn't N+1 the backend.
• Raise timeoutSeconds to the Kubernetes max (30s) so a slow tail doesn't immediately fail. Cheap mitigation.
• Async claim model for resources where transient over-allocation is tolerable: admit immediately, create the claim async, revoke if it turns out to be over. The compute repo's reason codes (Misconfigured, BackendUnavailable) already accommodate this shape; the design supports it.
• Internal webhook retry on backend transient failures before returning the user-visible 403. The error message already says "Please try again in a moment" — the webhook is asking the client to do work that should be done in the server.

Mitigation on the CLI side (this PR)

d8f7c96 adds is_quota_check_timeout / with_quota_check_retry and wraps every .create() site in the tunnel listen flow. Up to ~15s of backoff before propagating the error. Distinct from real quota exhaustion (which uses different wording — Insufficient quota — and is preserved as the friendly "exceeded" message). This makes the experience tolerable but it's a workaround; the right fix is in Milo.

Action for @drewr

If the Milo side analysis above looks plausible, this is worth a separate placeholder issue against whichever Milo repo owns the quota webhook + backend, including the operator-side mitigations as suggestions. Happy to draft that issue too if you confirm the diagnosis.

[drewr]

Status update

Picking up where the diagnosis-arc comment left off. Significant ground covered since the runtime-watch fix.

Landed since last update

Tunnel selection & resume UX

• ca4470f — `tunnel listen --id ` pins an existing tunnel and re-points its connector backend at the current iroh identity. Direct-API lookup; bypasses the connector filter that was hiding tunnels from list_active after identity rotation. get_active refactored to a direct fetch (no filter), shared summary_from_proxy helper extracted.
• a68d8ae — --endpoint is now optional. --id alone resumes verbatim from stored endpoint; --id + --endpoint must agree exactly or fail hard (preserving any URL already shared with others). lib::normalize_endpoint exposed so the CLI uses the same canonicalization as TunnelSummary.endpoint.
• 7de50c7 — arrow-key picker (inquire) when called with no flags, showing hostname → endpoint [label] rows. Adds `inquire = 0.9.4` to CLI deps.
• fe57b24 — picker uses a tracing_subscriber::reload::Handle + RAII guard to silence tracing while inquire is repainting (iroh log lines were splicing into the picker's first option line).
• cff37e7 — single-candidate auto-adopts with a one-liner instead of popping a picker that has no movement possible (terminal-cursor confusion).

Progress / verification

• c39d9ee — each progress checkpoint annotated with its underlying Kubernetes resource ([HTTPProxy/tunnel-…] / [Connector/datum-connect-…]). Lets a user pivot from a stuck step straight into datumctl describe ….
• d65ec4d — progress steps require observedGeneration ≥ metadata.generation for Ready. Closed the "Ready after 0 sec but data plane is dead" gap on resume-time spec patches (still relevant for any caller that does spec patch — even though we now skip the patch entirely, see below).
• 1ed969e — new `Verifying connectivity...` phase. Probes origin and proxy URL every 10s after controller conditions all flip Ready. New `--timeout` flag (default 10m) caps total setup. Distinguishes "your local server is down" from "edge still 503ing" in the failure summary. This is the visibility surface that exposed all the iroh-dial-latency findings below.

Lib hardening

• d8f7c96 — quota-check-timeout retry. Operator's 403 "took too long to be checked against your quota" is now retried with ~15s of backoff at every .create() in the listen flow (HTTPProxy, ConnectorAdvertisement, TrafficProtectionPolicy, Connector). Real quota exhaustion still surfaces the friendly "Quota limit exceeded" message; the carve-out is on the exact transient phrase.
• b7e9d6b — HeartbeatAgent::start_manual() skips the auto-fan-out across every accessible project. CLI tunnel-listen now heartbeats only the project the tunnel lives in. UI's start() is unchanged (it actually wants the multi-project model).
• 0311960 — update_project is idempotent at the lib boundary. Skips the HTTPProxy.spec and ConnectorAdvertisement.spec patches when the existing state already matches what we'd write. Comparison is on serde_json::Value for stable representation. This is hygiene for the upcoming factor-shared-tunnel-logic-into-lib work (cli + ui + future datumctl plugin all benefit automatically); the UI's Edit-without-changes case also stops causing data-plane churn for free.

Issues filed against other repos (all placeholders awaiting @drewr review)

Issue	Scope
datum-cloud/network-services-operator#174	Stale iroh DNS claim outlives its owning Connector — DeferredToOwner cites a UID that 404s in the API.
datum-cloud/network-services-operator#175	Narrowed today: fresh-tunnel HTTPProxy Programmed / ConnectorMetadataProgrammed blocked ~3min on EnvoyPatchPolicy has no status yet. Original conflation with the resume-side 5xx symptom is split out (next row).
datum-cloud/iroh-gateway#12	New today: variable multi-minute iroh dial latency from edge to a freshly-started listen node. Two contrasting runs (530ms vs 170s, same machine, 15min apart) as evidence. All control-plane and CLI-side causes are explicitly ruled out.
(PR-comment)	Quota 403 diagnosis — apiserver → Milo quota admission webhook → quota backend chain analysis with concrete mitigations on the Milo side. Not filed against a Milo repo yet, pending @drewr direction.

Known still-flaky behavior (mitigated, not fixed)

On a resume run, the proxy URL still occasionally returns 5xx for 1–3 minutes after Tunnel ready would have fired without the new verify phase. The verify phase makes this visible and waits for actual 2xx instead of false-positive-claiming Ready, so the user experience is "honest progress + slow first request" rather than "instant Ready + broken tunnel." The underlying iroh-gateway dial latency is tracked in iroh-gateway#12.

Test coverage on this branch

47 lib tests, all green. Notable new coverage:

• quota_check_timeout_classifier_matches_transient_403 — retry classifier + format carve-out.
• progress_pending_when_status_is_stale_for_current_generation — observedGeneration check.
• progress_step_carries_resource_label — every step backed by the right Kubernetes kind.
• http_proxy_spec_matches_* / advertisement_spec_matches_* — idempotency comparators across drift axes.
• start_manual_does_not_auto_enroll — heartbeat manual mode.
• listen_key_for_project_* (3 cases) — per-project key migration.

What's next

A separate work stream to factor shared tunnel-management logic out of cli/ and ui/ into the lib, so a future datumctl connect plugin can share the same authoritative implementation. The idempotency landing was a prerequisite for that — lib verbs are now the right shape to be the shared abstraction. Will be tracked separately when it kicks off.