FSFO observer provisioning and configuration

[pythianakhmadeev]

Change Description:

Provide an option to configure a separate machine as an Observer for a Fast-Start Failover configuration of the Data Guard setup.

Solution Overview:

Add a new parameter --config-observer to setup and configure an FSFO Observer on a dedicated host.

Test Commands:

Oracle 19c and 26ai Data Guard on FS and ASM as following (example for 19c):

export INSTANCE_IP_ADDR=
export STANDBY_IP_ADDR=
export FSFO_IP_ADDR=

# Data Guard provisioning
./install-oracle.sh \
  --instance-ip-addr ${INSTANCE_IP_ADDR} \
  --ora-edition EE \
  --ora-version 19 \
  --ora-swlib-bucket gs://pythian-gto-oracle-software/19c \
  --ora-disk-mgmt FS \
  --ora-db-name ora19 --ora-db-unique-name ora19p \
  --ora-data-mounts-json '[{"purpose":"software","blk_device":"/dev/disk/by-id/google-oracle-disk-1","name":"u01","fstype":"xfs","mount_point":"/u01","mount_opts":"nofail"},{"purpose":"data","blk_device":"/dev/disk/by-id/google-oracle-data-1","name":"u03","fstype":"xfs","mount_point":"/u03","mount_opts":"nofail"},{"purpose":"reco","blk_device":"/dev/disk/by-id/google-oracle-reco-1","name":"u04","fstype":"xfs","mount_point":"/u04","mount_opts":"nofail"}]' \
  --ora-data-destination "/u03/app/oracle/oradata" \
  --ora-reco-destination "/u04/app/oracle/fast_recovery_area" \
  --backup-dest "/u04/app/oracle/backups"

./install-oracle.sh \
  --instance-ip-addr ${STANDBY_IP_ADDR} \
  --primary-ip-addr ${INSTANCE_IP_ADDR} \
  --cluster-type DG \
  --ora-edition EE --ora-version 19 \
  --ora-swlib-bucket gs://pythian-gto-oracle-software/19c \
  --ora-disk-mgmt FS \
  --ora-db-name ora19 --ora-db-unique-name ora19s1 \
  --ora-data-mounts-json '[{"purpose":"software","blk_device":"/dev/disk/by-id/google-oracle-disk-1","name":"u01","fstype":"xfs","mount_point":"/u01","mount_opts":"nofail"},{"purpose":"data","blk_device":"/dev/disk/by-id/google-oracle-data-1","name":"u03","fstype":"xfs","mount_point":"/u03","mount_opts":"nofail"},{"purpose":"reco","blk_device":"/dev/disk/by-id/google-oracle-reco-1","name":"u04","fstype":"xfs","mount_point":"/u04","mount_opts":"nofail"}]' \
  --ora-data-destination "/u03/app/oracle/oradata" \
  --ora-reco-destination "/u04/app/oracle/fast_recovery_area" \
  --backup-dest "/u04/app/oracle/backups"

# FSFO installation and configuration
./install-oracle.sh \
  --config-observer \
  --instance-ip-addr ${FSFO_IP_ADDR} \
  --primary-ip-addr ${INSTANCE_IP_ADDR} \
  --cluster-type DG \
  --ora-edition CLIENT --ora-version 19 \
  --ora-swlib-bucket gs://pythian-gto-oracle-software/19c \
  --ora-disk-mgmt FS \
  --ora-db-name ora19 \
  --ora-data-mounts-json '[{"purpose":"software","blk_device":"/dev/disk/by-id/google-oracle-disk-1","name":"u01","fstype":"xfs","mount_point":"/u01","mount_opts":"nofail"}]'

Expected Result:

An FSFO Data Guard configuration is successfully deployed

[google-oss-prow]

Hi @pythianakhmadeev. Thanks for your PR.

I'm waiting for a google member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[mfielding]

For simplicity, we may want to avoid a separate software mount at all, given all we need to install is an Oracle client; can it just sit on the OS root mount?

[pythianakhmadeev]

This is not a straight forward thing to do, as far as I can see.

[mfielding]

Here are some suggestions to optimize the client-mode installation (for DG Observer) and avoid potential issues.

General Suggestions (for files not in the diff):

1. Optimize patch role for Client:
To support patching the client home without failing (due to missing database instance), we should modify roles/patch/tasks/main.yml and roles/patch/tasks/opatch_apply.yml:

• Skip database role checks (sqlplus / as sysdba) when ora_edition == 'CLIENT'.
• Skip srvctl stop/start home tasks for client homes.
• Skip SQL patching (datapatch) in opatch_apply.yml for client.

2. Avoid creating grid user for Client:
Currently, the grid user is still created because oracle_users in roles/common/defaults/main.yml is not modified. If we want to fully optimize, we should make grid user creation conditional. Note that if we do this, we also need to adjust oracle_dirs in roles/common/defaults/main.yml because oracle_inventory is currently configured to be owned by grid_user by default, which would fail if the user doesn't exist.

[mfielding]

We have verified the FSFO observer provisioning in client mode and found a few issues that need to be addressed:

1. Wallet Credential Case Sensitivity (Critical):
   The current code only adds uppercase credentials (${x^^}) to the wallet (e.g., ORCL). However, if the database name/SID is lowercase (e.g., orcl), the TNS alias and connection identifier are also lowercase. When the observer tries to connect using dgmgrl /@orcl, the wallet lookup fails with ORA-01017: logon denied.
   Fix: Update roles/dg-observer/tasks/main.yml to store both uppercase and lowercase credentials:

   echo -e "${OBSERVER_PWD}\n${OBSERVER_PWD}\n${WALLET_PWD}" | mkstore -wrl "{{ wallet_dir }}" -createCredential "${x^^}" "{{ dg_observer_user }}"
   echo -e "${OBSERVER_PWD}\n${OBSERVER_PWD}\n${WALLET_PWD}" | mkstore -wrl "{{ wallet_dir }}" -createCredential "${x,,}" "{{ dg_observer_user }}"

2. Wallet Password Conflict on Reruns:
   During incremental runs/testing, the random wallet password is regenerated. If the wallet directory already exists, mkstore fails to open it with the new password.
   Fix: Delete the existing wallet directory before recreating it in roles/dg-observer/tasks/main.yml:

   - name: Delete Wallet Directory if exists
     file:
       path: "{{ wallet_dir }}"
       state: absent

3. Shell Script Cleanup (Pipes vs Heredocs):
   The wallet creation task in roles/dg-observer/tasks/main.yml had a confusing mix of stdin pipes and indented heredocs (e.g., echo ... | mkstore ... <<EOF). Indented heredocs can also lead to terminator alignment bugs in bash if not handled carefully.
   Fix: Simplify the script to use standard piping only (as shown in the fix for issue 1).

4. Readiness Check Memory Requirement:
   The default memory readiness check requires 2.5 GB of memory. While this is appropriate for database nodes, it is excessive for dedicated observer nodes.
   Fix: Skip the memory check in roles/common/tasks/readiness_checks.yml when ora_edition is CLIENT:

   - name: readiness_check | Target memory >= {{ os_minimum_memory_mb }}
     assert:
       ...
     when:
       - managed_host_checks
       - not (ora_edition | default('EE') | lower == 'client')

[mfielding]

/ok-to-test

[mfielding]

/ok-to-test

[mfielding]

Nit: typo in --ora-disk-mgmt

[mfielding]

With the latest changes, I'd say:

Run the Data Guard Observer installation and configuration playbooks.

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mfielding, pythianakhmadeev

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [mfielding]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment