Skip to content

RHINENG-27056: move db-migration to a separate job #2236

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
TenSt wants to merge 1 commit into
base: master
Choose a base branch
from
+44 −16
Open

RHINENG-27056: move db-migration to a separate job #2236

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 14 additions & 2 deletions database_admin/entrypoint.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,20 @@
#!/bin/bash

set -e -o pipefail # stop on error

@sourcery-ai sourcery-ai Bot Jun 18, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue (bug_risk): Dropping set -e may mask failures in future commands before the migrate loop.

The retry loop correctly handles migrate failures, but without set -e any other command before or between attempts can fail silently and the script will keep running. To preserve fail-fast behavior while still allowing retries, consider restoring set -e and isolating the retry logic around migrate (for example, by wrapping migrate in a function whose exit code you control).

set -o pipefail

MIGRATION_FILES=file://./database_admin/migrations
MAX_RETRIES=${MIGRATION_MAX_RETRIES:-1}

echo "Running in $(pwd) as $(id)"
${GORUN:+go run} ./main${GORUN:+.go} migrate $MIGRATION_FILES
for attempt in $(seq 1 "$MAX_RETRIES"); do
echo "Migration attempt ${attempt}/${MAX_RETRIES}"
if ${GORUN:+go run} ./main${GORUN:+.go} migrate $MIGRATION_FILES; then
exit 0
fi
if [ "$attempt" -lt "$MAX_RETRIES" ]; then
echo "Migration failed, retrying..."
sleep 5
fi
done
echo "Migration failed after ${MAX_RETRIES} attempts"
exit 1
44 changes: 30 additions & 14 deletions deploy/clowdapp.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,23 +55,11 @@ objects:
podSpec:
image: ${IMAGE}:${IMAGE_TAG}
initContainers:
- name: db-migration
- name: check-for-db
image: ${IMAGE}:${IMAGE_TAG}
command:
- ./database_admin/entrypoint.sh
- ./database_admin/check-upgraded.sh
env:
- {name: LOG_LEVEL, value: '${LOG_LEVEL_DATABASE_ADMIN}'}
- {name: DB_DEBUG, value: '${DB_DEBUG_DATABASE_ADMIN}'}
- {name: GIN_MODE, value: '${GIN_MODE}'}
- {name: SHOW_CLOWDER_VARS, value: ''}
- {name: MANAGER_PASSWORD, valueFrom: {secretKeyRef: {name: patchman-engine-database-passwords,
key: manager-database-password}}}
- {name: LISTENER_PASSWORD, valueFrom: {secretKeyRef: {name: patchman-engine-database-passwords,
key: listener-database-password}}}
- {name: EVALUATOR_PASSWORD, valueFrom: {secretKeyRef: {name: patchman-engine-database-passwords,
key: evaluator-database-password}}}
- {name: VMAAS_SYNC_PASSWORD, valueFrom: {secretKeyRef: {name: patchman-engine-database-passwords,
key: vmaas-sync-database-password}}}
- {name: POD_CONFIG, value: '${DATABASE_ADMIN_CONFIG}'}
command:
- ./scripts/entrypoint.sh
Expand Down Expand Up @@ -313,6 +301,33 @@ objects:
requests: {cpu: '${CPU_REQUEST_EVALUATOR_USER_EVALUATION}', memory: '${MEM_REQUEST_EVALUATOR_USER_EVALUATION}'}

jobs:
- name: db-migration
completions: 1
parallelism: 1
activeDeadlineSeconds: ${{MIGRATION_TIMEOUT}}
podSpec:
image: ${IMAGE}:${IMAGE_TAG}
command:
- ./database_admin/entrypoint.sh
env:
- {name: MIGRATION_MAX_RETRIES, value: '3'}

@sourcery-ai sourcery-ai Bot Jun 18, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: Consider making MIGRATION_MAX_RETRIES configurable via a template parameter instead of hardcoding '3'.

Align this with how MIGRATION_TIMEOUT and other resource values are configured, so the retry policy can be tuned per environment without editing the manifest directly.

Suggested implementation:

      podSpec:
        image: ${IMAGE}:${IMAGE_TAG}
        command:
          - ./database_admin/entrypoint.sh
        env:
        - name: MIGRATION_MAX_RETRIES
          value: ${{MIGRATION_MAX_RETRIES}}

To fully implement the suggestion, you will also need to:

  1. Declare a MIGRATION_MAX_RETRIES template parameter in this ClowdApp (or the surrounding template/Helm chart), following the same pattern used for MIGRATION_TIMEOUT (e.g., add it to the parameters/envTemplate section with a default value of 3).
  2. Ensure any environment-specific overlays (e.g., dev/stage/prod) can override MIGRATION_MAX_RETRIES so it can be tuned per environment without editing this manifest directly.

- {name: LOG_LEVEL, value: '${LOG_LEVEL_DATABASE_ADMIN}'}
- {name: DB_DEBUG, value: '${DB_DEBUG_DATABASE_ADMIN}'}
- {name: GIN_MODE, value: '${GIN_MODE}'}
- {name: SHOW_CLOWDER_VARS, value: ''}
- {name: MANAGER_PASSWORD, valueFrom: {secretKeyRef: {name: patchman-engine-database-passwords,
key: manager-database-password}}}
- {name: LISTENER_PASSWORD, valueFrom: {secretKeyRef: {name: patchman-engine-database-passwords,
key: listener-database-password}}}
- {name: EVALUATOR_PASSWORD, valueFrom: {secretKeyRef: {name: patchman-engine-database-passwords,
key: evaluator-database-password}}}
- {name: VMAAS_SYNC_PASSWORD, valueFrom: {secretKeyRef: {name: patchman-engine-database-passwords,
key: vmaas-sync-database-password}}}
- {name: POD_CONFIG, value: '${DATABASE_ADMIN_CONFIG}'}
resources:
limits: {cpu: '${CPU_LIMIT_DATABASE_ADMIN}', memory: '${MEM_LIMIT_DATABASE_ADMIN}'}
requests: {cpu: '${CPU_REQUEST_DATABASE_ADMIN}', memory: '${MEM_REQUEST_DATABASE_ADMIN}'}

- name: vmaas-sync
activeDeadlineSeconds: ${{JOBS_TIMEOUT}}
schedule: ${VMAAS_SYNC_SCHEDULE}
Expand Down Expand Up @@ -767,6 +782,7 @@ parameters:
- {name: CLEAN_AAD_SUSPEND, value: 'false'} # Disable cronjob execution

# Database admin
- {name: MIGRATION_TIMEOUT, value: '7200'} # 2h timeout for db-migration job
- {name: LOG_LEVEL_DATABASE_ADMIN, value: debug}
- {name: DB_DEBUG_DATABASE_ADMIN, value: 'false'}
- {name: CPU_LIMIT_DATABASE_ADMIN, value: 100m}
Expand Down
Loading