Skip to content

Support IPv6 Secondary Interface in shiftstack-qa automation#17

Open
tusharjadhav3302 wants to merge 3 commits into
mainfrom
support_ipv6_secondary_OSPRH-6486
Open

Support IPv6 Secondary Interface in shiftstack-qa automation#17
tusharjadhav3302 wants to merge 3 commits into
mainfrom
support_ipv6_secondary_OSPRH-6486

Conversation

@tusharjadhav3302

@tusharjadhav3302 tusharjadhav3302 commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add create_ipv6_secondary_networks.yml prepare task that creates IPv6 networks
    (slaac + dhcpstateless), subnets, and router for secondary worker interfaces
  • Add configure_ipv6_secondary day2ops procedure that applies DHCP kernel arg
    MachineConfig, patches CNO with macvlan additionalNetworks, deploys test pods,
    and verifies pod-to-pod IPv6 connectivity
  • Fix naming mismatch: validation now reads secondary_ip_protocol instead of
    non-existent network_secondary
  • Add osp_verification_ipv6_secondary.yaml job definition (4-stable) for the scenario
  • Add IPv6 secondary network config block to configs/global.yml

Details

Prepare phase:

  • Creates 2 IPv6 networks with different RA modes (slaac, dhcpv6-stateless)
  • Creates IPv6 subnets with appropriate RA/address modes
  • Creates a router and attaches both subnets
  • Registers resources for cleanup tracking

Install phase:

  • Adds additionalNetworkIDs to install-config worker platform block when
    secondary_ip_protocol == 'ipv6', so Nova pre-creates IPv6 ports on workers

Post-install (day2ops procedure):

  • Applies 05-worker-kernelarg-dhcp MachineConfig for ip=dhcp,dhcp6
  • Waits for MCP update and cluster health
  • Discovers IPv6 interfaces on workers via oc adm node-logs
  • Patches CNO network.operator with macvlan additionalNetworks
  • Deploys hello-openshift pods with IPv6 network annotations
  • Disables port security on worker IPv6 ports (required for macvlan)
  • Verifies pod-to-pod connectivity on both slaac and dhcpstateless networks
  • Validates external reachability via oc debug node/ from worker nodes

Cleanup:

  • Dedicated cleanup_ipv6_secondary.yml detaches subnets, removes router and networks
  • Integrated into cleanup stage (runs before cluster destruction)

Future Scope (not in this PR)

  • ci-framework-jobs integration: Add periodic pipeline scenario file
    (scenarios/baremetal/shiftstack/tests-ipv6-secondary.yml) pointing to the
    new job definition for automated Zuul scheduling.
  • IPv6 VM connectivity testing: Deploy a standalone OpenStack VM attached to the
    IPv6 secondary networks for full VM-to-pod and pod-to-VM validation.
    Deferred because it requires admin-only OpenStack operations (image upload, flavor creation).
  • Worker scaleup re-verification: After scaling up workers, re-scale IPv6 deployments
    and re-check connectivity.

Test Plan

  • ansible-lint passes with production profile
  • ci-framework-testproject Zuul run with osp_verification_ipv6_secondary.yaml
  • IPv6 secondary networks created in OpenStack
  • Workers have IPv6 interfaces after MachineConfig application
  • NetworkAttachmentDefinition CRs created by CNO
  • Pods can communicate over IPv6 secondary interfaces (same node + different nodes)
  • External reachability validated via oc debug node/

Jira

https://redhat.atlassian.net/browse/OSPRH-6486

@tusharjadhav3302 tusharjadhav3302 force-pushed the support_ipv6_secondary_OSPRH-6486 branch 2 times, most recently from 52bb7ae to 80ab1dc Compare June 29, 2026 15:13
@tusharjadhav3302 tusharjadhav3302 added the ready-for-review PR is ready for code review label Jun 29, 2026

- name: Build pod name to IPv6 address mapping
ansible.builtin.set_fact:
ipv6_pod_ips: "{{ ipv6_pod_ips | default({}) | combine({item.item: item.stdout}) }}"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

default({}) only fires when the variable is undefined. Since this file is included in a loop (configure_ipv6_secondary.yml loops over projects), set_fact persists ipv6_pod_ips across iterations - the second network would inherit the first's pod-IP mappings, mixing IPs across namespaces.

The legacy IR plugin (check_ipv6_connectivity.yml:10-11) explicitly resets both variables before each iteration:

- set_fact:
    pods_names: []
    pods_ips: []

Adding a similar set_fact: ipv6_pod_ips: {} reset at the top of this file would fix it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added set_fact: ipv6_pod_ips: {} at the top to explicitly reset before each project loop, matching the IR plugin pattern.

kubernetes.core.k8s_info:
kubeconfig: "{{ kubeconfig }}"
api_version: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This k8s_info query has no namespace filter - it returns all NADs cluster-wide. The == check on the until assertion would fail if any other NADs exist (Multus defaults, SR-IOV operator, other tests).

Note: this is the same pattern from the legacy IR plugin (configure-ipv6-networks.cno.yml:64), so it's a ported behavior. Still worth fixing since the migration is an opportunity - consider adding a namespace filter per project, or using a subset check.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added namespace: "{{ item }}" filter and switched to per-namespace loop verification.

- name: Discover IPv6 interfaces on first worker
ansible.builtin.shell: |
set -o pipefail && \
oc adm node-logs {{ ocp_workers[0] }} | \

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: parsing oc adm node-logs journal output for interface names is fragile - log format changes between OCP versions could break the sed regex. verify_ipv6_external_reachability.yml already uses oc debug node/ successfully - the same approach (ip -6 addr show scope global) would query live interface state directly.

Also, this discovers only on ocp_workers[0] - if NIC naming differs across workers, the macvlan config could reference a wrong interface on other nodes.

Note: the legacy IR plugin uses the same approach (configure-ipv6-networks.cno.yml:25), so this is a faithful port - but the migration is an opportunity to improve it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rewritten to use oc debug node/ + ip -6 addr show scope global on ALL workers instead of parsing logs on a single worker. The new approach:

  1. Queries live interface state directly (version-independent, no log format dependency)
  2. Discovers on every worker, not just the first one
  3. Asserts NIC naming is consistent across all workers before proceeding
  4. Fails with a clear per-worker mapping if naming diverges

Added timeout 120 to prevent hangs on unresponsive nodes.

cloud: "{{ user_cloud }}"
state: absent
name: "{{ item.net_name }}"
loop: "{{ ipv6_secondary_networks.networks }}"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: the router and subnet operations above correctly use registered_resources from the resources file, but network deletion uses ipv6_secondary_networks.networks (static config). Same pattern as the legacy (remove_ipv6_resources.yml:38) - not blocking, but using the registered IDs would be more robust.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Network deletion now uses registered_resources.ipv6_secondary_network_ids (from the resources file) instead of the static config variable, matching the pattern used by the router and subnet operations above it. This makes cleanup self-contained and independent of whether ipv6_secondary_networks is loaded.

type: "{{ installcfg_worker_flavor }}"
serverGroupPolicy: "{{ installcfg_worker_servergrouppolicy }}"
{%- if ocp_deployment_topology.secondary_ip_protocol == 'ipv6' +%}
additionalNetworkIDs: {{ ipv6_secondary_network_ids | to_json }}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: ipv6_secondary_network_ids is set via set_fact in the prepare play (localhost). This works when deploy_installer_host: false (the default) because installer=localhost. But with deploy_installer_host: true (used by 5 existing job definitions), the variable would be undefined here.

The IDs are already saved to the resources file - loading them back in the install role would make this portable.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The install role now loads ipv6_secondary_network_ids from the resources file via include_vars (same pattern as proxy_install_config.yml) before rendering the template. This makes it portable with deploy_installer_host: true

@tusharjadhav3302 tusharjadhav3302 added enhancement New feature or request and removed ready-for-review PR is ready for code review labels Jul 8, 2026
tusharjadhav3302 and others added 3 commits July 8, 2026 13:02
…PRH-6486

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Development

Successfully merging this pull request may close these issues.

2 participants