fix(metrics): cap distinct HTTP path labels per container

[mayankpande88]

Problem

Prometheus in nudgebee-agent on the Rackspace cluster is being OOMKilled (7 restarts against an 8Gi limit). job=nudgebee-node-agent is 447k of ~690k head series (65%).

Two independent drivers. This PR addresses the second.

1. Destination IP churn — already fixed, not shipped there. actual_destination was a raw IP:port, so sqs.us-east-1.amazonaws.com alone minted 331 distinct series for one logical destination. #280 collapses this to workload identity and landed in v0.1.1; the cluster runs 0.1.0. Fixed by upgrading, no code change needed.

2. Scanner-driven path cardinality — unfixed until now. The two internet-facing ingress-nginx pods had accumulated 10,349 distinct path values, 9,601 of them 404s:

/axds.php  /HNAP1  /wp-content/plugins/hellopress/wp_filemanager.php
/biufile.php  /faze.php  /this_is_a_new_hello_world.php  ...

normalizeHttpPath cannot collapse these — they are genuinely distinct literals, not parameterized routes. path is the only L7 label an outside party controls, and nothing bounded it, so container_http_requests_total grew a permanent series per probe for as long as the scanning continued.

Change

A per-container common.PathLimiter in front of the path label:

• Paths seen before the cap always pass through — a steady application keeps reporting its real routes, and scanner traffic arriving later cannot displace them.
• Paths first observed after the cap fills collapse to {other}, so the request count stays correct while the series count stops growing.
• Normalization runs first, so /api/users/{id} consumes one slot, not one per ID.
• Default 1000 via MAX_HTTP_PATHS_PER_CONTAINER, well above what a real application serves. 0 restores the previous unbounded behaviour.
• One-shot warning when a container hits the cap, so this is visible rather than silent.

Latency histograms are untouched — they carry no path label.

Why common/ and not containers/

CI runs go test $(go list ./... | grep -v '/containers$'). The containers package is excluded because NVML's dlsym aborts the test binary on non-GPU runners, so a test placed there would never execute. The logic lives in common/, which CI does test, and L7Stats just holds one.

Testing

6 tests, race-clean, verified in a Linux container (containers does not compile on macOS):

• admits up to the cap, then collapses to {other}
• already-admitted paths keep passing after the cap fills
• 0 means unlimited
• empty path (non-HTTP protocols, invalid UTF-8) neither consumes a slot nor gets rewritten
• concurrent admission never overshoots the cap — otherwise the bound is not a bound
• nil receiver degrades to unbounded instead of panicking in the L7 hot path
• one container filling its cap does not silence another's routes

Full CI-equivalent run on Linux: gofmt -l clean, go vet ./... clean, all non-containers packages pass, go build -mod=readonly . succeeds.

Rollout

Upgrading Rackspace off 0.1.0 is the larger win of the two (it removes the 255k histogram-bucket series that come purely from IP churn). This cap is what stops the counter side regrowing afterward. Watch the chart pin on that upgrade: k8s-agent charts <0.1.17 hardcode command: ["coroot-node-agent"], so an image-only bump gives RunContainerError.

[gemini-code-assist]

Code Review

This pull request introduces a PathLimiter to bound the number of distinct HTTP path label values per container, preventing cardinality explosion from scanner traffic. It also adds a configuration flag MaxHttpPathsPerContainer and integrates the limiter into L7Stats. Feedback suggests making the zero value of PathLimiter usable by lazily initializing the seen map to prevent potential panics when the struct is zero-initialized.

[gemini-code-assist]

In Go, it is idiomatic and highly recommended to make the zero value of a struct usable. Currently, if PathLimiter is initialized as a zero value (e.g., var limiter common.PathLimiter), calling Limit will panic on the first write because the seen map is nil. We can easily make the zero value usable by lazily initializing the seen map under the write lock and safely checking for nil in the read lock path.

	l.mu.RLock()
	var seen bool
	if l.seen != nil {
		_, seen = l.seen[path]
	}
	full := l.full
	l.mu.RUnlock()
	if seen {
		return path
	}
	if full {
		return OtherPath
	}

	l.mu.Lock()
	defer l.mu.Unlock()
	if l.seen == nil {
		l.seen = make(map[string]struct{})
	}
	// Re-check: another request may have admitted this path, or filled the cap,
	// while we waited for the write lock.
	if _, seen := l.seen[path]; seen {
		return path
	}