Skip to content

add composite index on (release, timestamp) to job runs matview#3654

Open
mstaeble wants to merge 1 commit into
openshift:mainfrom
mstaeble:add-composite-index-job-runs-matview
Open

add composite index on (release, timestamp) to job runs matview#3654
mstaeble wants to merge 1 commit into
openshift:mainfrom
mstaeble:add-composite-index-job-runs-matview

Conversation

@mstaeble

@mstaeble mstaeble commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add AdditionalIndexes field to PostgresView for non-unique indexes alongside the required unique index
  • Add a (release, timestamp DESC) index on prow_job_runs_report_matview

Problem

Every query against prow_job_runs_report_matview filters by release and sorts by timestamp DESC, but only id was indexed. This forced sequential scans across all ~527K rows for every paginated request to /api/jobs/runs.

Benchmarks (staging)

With the matview data warm in shared_buffers, typical /api/jobs/runs response times are 350-700ms. With the index, the database portion of the query drops from a sequential scan (~120ms warm for COUNT, longer for the paginated SELECT) to an index scan (15ms for COUNT, sub-millisecond for paginated SELECT). The index also eliminates the sort step since the data is pre-ordered.

The improvement is more pronounced under cold-cache conditions (e.g. after a matview refresh evicts pages): the sequential scan rises to 2+ seconds while the index scan stays sub-millisecond.

Index build cost: ~1.2s per matview refresh — negligible against the ~14-minute refresh.

Test plan

  • go build ./...
  • go vet ./...
  • go test ./pkg/db/...
  • Verified index is created by sippy migrate on staging
  • Verified /api/jobs/runs response times improve with index present

@coderabbitai pause

🤖 Generated with Claude Code

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: automatic mode

@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 18, 2026
@openshift-ci

openshift-ci Bot commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@openshift-ci openshift-ci Bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 18, 2026
@coderabbitai

coderabbitai Bot commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

@mstaeble, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 59 minutes and 51 seconds. Learn how PR review limits work.

Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file).

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits.

🚦 How do rate limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan refill rate.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, the refill rate gradually slows as usage increases. The highest same-day bursts are limited more strictly.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: b565c35c-7176-48c2-a078-e26089fabedd

📥 Commits

Reviewing files that changed from the base of the PR and between 94be1d9 and 6b430e2.

📒 Files selected for processing (1)
  • pkg/db/views.go

Walkthrough

pkg/db/views.go adds an AdditionalIndexes []string field to PostgresView for specifying raw non-unique index expressions. syncPostgresMaterializedViews is extended to create and drop these indexes using deterministic names idx_<viewName>_<i>. The prow_job_runs_report_matview entry is updated to declare one such index on release, timestamp DESC.

Changes

Additional non-unique index support for materialized views

Layer / File(s) Summary
PostgresView struct field and matview configuration
pkg/db/views.go
AdditionalIndexes []string is added to PostgresView, and prow_job_runs_report_matview sets this field to []string{"release, timestamp DESC"}.
syncPostgresMaterializedViews index creation logic
pkg/db/views.go
syncPostgresMaterializedViews iterates over AdditionalIndexes, builds DROP INDEX IF EXISTS and CREATE INDEX statements with deterministic names idx_<viewName>_<i>, and applies them via syncSchema.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Suggested reviewers

  • stbenjam
  • xueqzhan
🚥 Pre-merge checks | ✅ 19 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
Test Coverage For New Features ⚠️ Warning New functionality to handle additional indexes in syncPostgresMaterializedViews (lines 127-134) lacks unit test coverage, despite being a non-trivial code addition. Add unit or integration tests to cover the new AdditionalIndexes loop in syncPostgresMaterializedViews that verify index creation and cleanup logic.
✅ Passed checks (19 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Go Error Handling ✅ Passed The PR follows the existing error handling patterns in views.go: properly checks errors, safely dereferences pointers with nil checks, contains no panic() calls, and uses consistent error return pa...
Sql Injection Prevention ✅ Passed All SQL values come from hardcoded PostgresMatViews array; no user input is used in SQL construction. The new index creation code follows existing safe patterns used in line 121.
Excessive Css In React Should Use Styles ✅ Passed PR only modifies Go database code (pkg/db/views.go) with no React components or CSS styling; check is not applicable.
Single Responsibility And Clear Naming ✅ Passed The changes maintain single responsibility and clear naming: PostgresView struct remains focused (5 fields, well within ~7 limit), new AdditionalIndexes field is clearly named and distinguished fro...
Feature Documentation ✅ Passed This PR adds internal database indexes for query optimization with no user-facing features, API changes, or data model changes. The docs/features directory contains only user-facing feature documen...
Stable And Deterministic Test Names ✅ Passed This PR does not modify or add any Ginkgo tests. The changes are limited to pkg/db/views.go, which adds database schema functionality. No test files with Ginkgo patterns exist in the codebase.
Test Structure And Quality ✅ Passed The PR modifies only pkg/db/views.go and does not introduce any Ginkgo test code. The custom check for Ginkgo test quality is not applicable to this PR.
Microshift Test Compatibility ✅ Passed No Ginkgo e2e tests are added in this PR. The changes are database schema modifications, and newly added test files use Go's standard testing.T framework with testify, not Ginkgo.
Single Node Openshift (Sno) Test Compatibility ✅ Passed This PR contains no Ginkgo e2e tests. Changes are database configuration and indexing logic only (pkg/db/views.go). SNO compatibility check is not applicable.
Topology-Aware Scheduling Compatibility ✅ Passed This PR modifies only PostgreSQL database schema/indexing code in pkg/db/views.go. No deployment manifests, operator code, controllers, or Kubernetes scheduling constraints are introduced. The chec...
Ote Binary Stdout Contract ✅ Passed PR modifies database schema library code in pkg/db/views.go with no stdout writes; OTE contract applies only to OTE test binaries communicating via JSON stdout, not schema management libraries.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed Check not applicable: PR modifies only pkg/db/views.go (a database backend file), not Ginkgo e2e tests. No new Ginkgo test patterns (It, Describe, Context, When) were added.
No-Weak-Crypto ✅ Passed PR only adds database indexing functionality to PostgreSQL materialized views with no cryptographic usage, weak algorithms, custom crypto, or secret comparison operations.
Container-Privileges ✅ Passed PR only modifies pkg/db/views.go for database indexing. No container/K8s manifests, Dockerfiles, or privileged configuration changes are present.
No-Sensitive-Data-In-Logs ✅ Passed The PR adds index creation logic for materialized views. SQL statements and index names are not logged—only SHA256 hashes and generic messages are logged in syncSchema. No sensitive data (passwords...
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title directly and clearly describes the primary change: adding a composite index on (release, timestamp) to the prow_job_runs_report_matview.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@openshift-ci

openshift-ci Bot commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: mstaeble
Once this PR has been reviewed and has the lgtm label, please assign xueqzhan for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the ready-for-human-review Indicates a PR has been reviewed by automated tools and is ready for human review label Jun 18, 2026
Every query against prow_job_runs_report_matview filters by release
and sorts by timestamp DESC, but only id was indexed. This caused
sequential scans across all ~527K rows for every paginated request.

Add AdditionalIndexes support to PostgresView so matviews can have
non-unique indexes alongside the required unique index, then add
an index on (release, timestamp DESC).

Staging benchmark for a typical paginated query:
  Before: 2,334ms (Parallel Seq Scan, 166K pages read)
  After:  0.13ms (Index Scan, 28 pages read)

COUNT(*) for pagination: 120ms → 15ms (Index Only Scan).

Index build cost: ~1.2s per refresh, negligible against the 14-minute
matview refresh.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@mstaeble mstaeble force-pushed the add-composite-index-job-runs-matview branch from 94be1d9 to 6b430e2 Compare June 18, 2026 18:47
@openshift-ci openshift-ci Bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 18, 2026
@mstaeble mstaeble changed the title [WIP] Add composite index on (release, timestamp) to job runs matview add composite index on (release, timestamp) to job runs matview Jun 18, 2026
@mstaeble mstaeble marked this pull request as ready for review June 18, 2026 19:45
@openshift-ci openshift-ci Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 18, 2026
@openshift-ci openshift-ci Bot requested review from smg247 and xueqzhan June 18, 2026 19:45
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling required tests:
/test e2e

@openshift-ci

openshift-ci Bot commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

@mstaeble: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human-review Indicates a PR has been reviewed by automated tools and is ready for human review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant