fix: partition metrics_anomaly_score training stats by dimension

[sravankumarkunadi]

Fixes elementary-data/elementary#1729, and addresses the training_avg half of elementary-data/elementary#2172.

(Both issues are filed on the elementary repo, but the code they describe lives here.)

Problem

In models/edr/data_monitoring/anomaly_detection/metrics_anomaly_score.sql, all five window functions partition by metric_name, full_table_name, column_name only:

avg(metric_value) over (
    partition by metric_name, full_table_name, column_name
    order by bucket_start asc
    rows between unbounded preceding and current row
) as training_avg,

dimension and dimension_value are selected in the CTE and in the group by, but they're missing from the partitions. For dimension-based metrics that pools every dimension value into one training set, so training_avg, training_stddev, training_set_size, training_start and training_end describe the table as a whole rather than the individual dimension value.

The distortion scales with how uneven the dimension values are. Using the example from #2172 — daily counts per row_type:

	D1	D2	D3	D4
X	1000	1001	1010	1002
Y	200	200	201	7000
Z	10	10	9	10

Before: every row is scored against a pooled average of ~400. X, which has been flat at ~1000 for its entire history, sits far above that mean and reads as anomalous on every bucket. The genuine Y spike at D4 is diluted by the same pooling.

After: X is scored against X's history, Y against Y's, Z against Z's. X reads as stable, and Y's D4 spike stands out against its own ~200 baseline.

This matches the reported symptom in #2172 — latest_metric_value correct when grouped by the dimension, but training_avg far below that dimension's real historical values.

Change

Added dimension, dimension_value to the partition by of all five window functions.

This brings the model in line with macros/edr/data_monitoring/anomaly_detection/get_anomaly_scores_query.sql, which already partitions the equivalent statistics by metric_name, full_table_name, column_name, dimension, dimension_value (see partition_by_keys, line 62). The model was simply never updated to match.

Including dimension as well as dimension_value also handles the case @wbarth11 raised in #1729, where the same value string appears under two different dimensions.

Scope

• Non-dimension metrics are unaffected. dimension and dimension_value are null for them, and partition by treats nulls as a single group, so those rows stay in one partition exactly as before.
• The test path is unaffected. get_anomaly_scores_query already partitioned correctly; this only changes the metrics_anomaly_score view, which is what users query directly and what feeds anomaly_threshold_sensitivity.
• sqlfmt (v0.29.0, matching .pre-commit-config.yaml) passes on the changed file.

Testing

Verified by inspection and sqlfmt. I don't have a warehouse to run the integration suite against — the existing integration_tests/tests/test_dimension_anomalies.py cases exercise the test-time path via get_anomaly_scores_query, so they don't cover this view. Happy to add coverage for metrics_anomaly_score if you can point me at the right place for it.

Summary by CodeRabbit

• Bug Fixes
  • Improved anomaly detection accuracy by computing historical training statistics separately for each metric dimension and dimension value.
  • This prevents unrelated dimensions from affecting anomaly scores, resulting in more reliable scoring across varied metrics.

[github-actions]

👋 @sravankumarkunadi
Thank you for raising your pull request.
Please make sure to add tests and document all user-facing changes.
You can do this by editing the docs files in the elementary repository.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: fc40fe66-3817-470d-8535-8ec06257ae7a

📥 Commits

Reviewing files that changed from the base of the PR and between e94c119 and db1d98a.

📒 Files selected for processing (1)

• models/edr/data_monitoring/anomaly_detection/metrics_anomaly_score.sql

🚧 Files skipped from review as they are similar to previous changes (1)

• models/edr/data_monitoring/anomaly_detection/metrics_anomaly_score.sql

---

📝 Walkthrough

Walkthrough

The anomaly score model now calculates training statistics and window boundaries separately for each combination of metric, table, column, dimension, and dimension value.

Changes

Anomaly training partitioning

Layer / File(s)	Summary
Expand training window partitions models/edr/data_monitoring/anomaly_detection/metrics_anomaly_score.sql	Training averages, standard deviations, set sizes, and start/end boundaries now partition by dimension and dimension_value in addition to the existing metric, table, and column fields.

Estimated code review effort: 2 (Simple) | ~10 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: partitioning anomaly score training stats by dimension.
Linked Issues check	✅ Passed	The PR satisfies #1729 by adding dimension and dimension_value to the training-statistics partitions in metrics_anomaly_score.sql.
Out of Scope Changes check	✅ Passed	The changes are narrowly scoped to the requested anomaly-scoring partition logic with no unrelated edits.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[sravankumarkunadi]

@arbiv kindly review the PR and please let me know if anything should be adjusted.