Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 26 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -454,7 +454,7 @@ of the three may veto a candidate destination.

The cache is invalidated each engine cycle.

Future: NUMA, CPU feature flags, flavor extra specs, soft-rule
Future: NUMA, flavor extra specs, soft-rule
penalties in the planner -
https://docs.openstack.org/nova/latest/user/server-groups.html

Expand Down Expand Up @@ -798,3 +798,28 @@ Engine notification listener updates cooldown/quarantine state
4. Hard timeout on migration polling
5. Post-flight verification (host + status)
6. Idempotent: pre-flight catches duplicate or stale tasks


## CPU compatibility gate

`[engine] require_cpu_compatibility` defaults to false and is independent
of the Placement claims gate. `PlacementClient.fetch_cpu_traits(hosts)`
reads `HW_CPU_*` traits using Placement microversion 1.6. The engine
resolves aggregate/AZ scope before fetching the union once per cycle.

`ConstraintChecker` requires source traits to be a subset of destination
traits after availability and claims checks, before server-group checks.
All four movers inherit the check. Pair verdicts and trait data reset
each cycle. Missing data fails closed. A known empty source set warns
and passes when the destination also has known data.

Both snapshot entry points capture `placement/cpu_traits.json` when
enabled. `ReplayPlacementClient` reads it without network access.
Missing or invalid snapshots fail closed. Claims remain unrecorded,
so CPU replay requires `enforce_placement_claims = false`.

The host-level rule is approximate. `HW_CPU_*` includes hyperthreading
as well as mapped instruction flags. Current host configuration can
differ from a running guest's retained CPU model. Nova remains the
authoritative migration compatibility check. See the configuration
reference for behavior and limitations.
53 changes: 53 additions & 0 deletions docs/configuration/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -43,3 +43,56 @@ commented example. Load-time invariants:
- ``imbalance_query`` must return per-host values in [0, 1] - enforced
at runtime by the scorer, which skips the policy for the cycle (with
an error logged) on out-of-range data.


CPU compatibility
-----------------

Set ``[engine] require_cpu_compatibility = true`` to require a
destination's Placement ``HW_CPU_*`` traits to contain every CPU trait
reported for the source host. The default is false. This option is
independent of ``enforce_placement_claims``.

The shared constraint checker applies this rule to spread, pack,
disabled-host evacuation, and affinity repair. Checks run after host
availability and capacity claims, before server-group checks. A DEBUG
message names the rejected host pair and missing traits.

The engine lists Placement providers and reads traits once per in-scope
host per cycle, deduplicating overlapping aggregates and filtering to
the configured availability zone. The service user needs
``placement:resource_providers:list`` and
``placement:resource_providers:traits:list`` access. The traits endpoint
uses Placement microversion 1.6. Provider names must match the Nova host
names used by the engine. Missing source or destination data blocks a
move. Any traits fetch failure blocks all moves for that cycle.

A successful response containing no ``HW_CPU_*`` traits is different
from unavailable data. An empty source set permits moves to destinations
with known data and logs a warning once per cycle that the check is
blind for that source.

This is a host-level approximation, not a per-instance CPU model check.
Nova's libvirt driver reports host features for host-model and
host-passthrough, and a union of configured models for custom mode.
The prefix also includes ``HW_CPU_HYPERTHREADING``, so the check can
reject moves for more than instruction-set differences. It can reject
moves Nova would accept and miss incompatibilities involving unmapped
flags or guests retaining an older CPU model after reconfiguration.
Nova's migration pre-check remains authoritative.

Producer source:
`Nova libvirt CPU trait reporting <https://github.com/openstack/nova/blob/stable/2025.2/nova/virt/libvirt/driver.py>`_.
API source:
`Placement provider traits <https://github.com/openstack/placement/blob/stable/2025.2/placement/handlers/trait.py>`_.

When the option is enabled, both ``kronos-record`` and engine SIGUSR1
snapshots write ``placement/cpu_traits.json``, a mapping from host names
to sorted trait lists. Failed collection writes an empty mapping.
Replay reads this file without contacting Placement. Missing files in
older snapshots fail closed when CPU checking is enabled.

For CPU compatibility replay, enable ``require_cpu_compatibility`` and
set ``enforce_placement_claims = false``. Capacity claims are not
recorded by the existing snapshot format. Leaving that gate enabled
pauses replay planning with an explicit error in the logs.
23 changes: 21 additions & 2 deletions docs/configuration/kronos.conf.sample
Original file line number Diff line number Diff line change
Expand Up @@ -292,6 +292,11 @@
# aggregate. (boolean value)
#enforce_placement_disk = false

# Require destination Placement HW_CPU_* traits to contain all source traits.
# Missing data blocks moves. Empty source sets warn and pass. Independent of
# enforce_placement_claims. Nova still checks migrations. (boolean value)
#require_cpu_compatibility = false


[executor]

Expand Down Expand Up @@ -387,6 +392,21 @@
# Log requests to multiple loggers. (boolean value)
#split_loggers = false

# An OpenSSL cipher string to set the allowed ciphers for TLS connections. If
# not specified, the default ciphers from the underlying OpenSSL library are
# used. (string value)
#tls_ciphers = <None>

# The minimum TLS version to require for TLS connections. When not specified,
# the behavior is undefined and defers to the system cryptographic policy.
# (string value)
# Possible values:
# <None> - None - the absence of a setting, defers to the system cryptographic
# policy
# 1.2 - TLS 1.2 - enforce minimum TLS version 1.2
# 1.3 - TLS 1.3 - enforce minimum TLS version 1.3
#tls_min_version = <None>

# The default service_type for endpoint URL discovery. (string value)
#service_type = <None>

Expand Down Expand Up @@ -584,8 +604,7 @@
# (boolean value)
# This option is deprecated for removal.
# Its value may be silently ignored in the future.
# Reason: Hostname verification should remain enabled once operators have
# completed the migration.
# Reason: Verification is always enabled now
#ssl_enforce_hostname_verification = true

# DEPRECATED: Global toggle for enforcing the OpenSSL FIPS mode. This feature
Expand Down
1 change: 1 addition & 0 deletions etc/kronos/kronos.conf.sample
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
# debug = false

[engine]
require_cpu_compatibility = false
# Seconds between policy evaluation cycles.
# evaluation_interval = 60

Expand Down
26 changes: 26 additions & 0 deletions kronos/clients/placement.py
Original file line number Diff line number Diff line change
Expand Up @@ -206,3 +206,29 @@ def _fetch_usages(self, rp_id: str) -> dict[str, int]:
for rc, used in raw.items()
if rc in TRACKED_RESOURCE_CLASSES
}

def fetch_cpu_traits(self, hosts: set[str]) -> dict[str, frozenset[str]]:
if not hosts:
return {}
try:
traits: dict[str, frozenset[str]] = {}
for rp in self._placement.resource_providers():
if rp.name not in hosts:
continue
response = self._placement.get(
f"/resource_providers/{rp.id}/traits", microversion="1.6",
)
response.raise_for_status()
raw = response.json()["traits"]
if not isinstance(raw, list) or not all(
isinstance(trait, str) for trait in raw
):
raise ValueError(f"Invalid CPU traits response for {rp.name}")
traits[rp.name] = frozenset(
trait for trait in raw if trait.startswith("HW_CPU_")
)
return traits
except Exception as exc:
raise PlacementClientError(
reason=f"Failed to fetch CPU traits: {exc}",
) from exc
5 changes: 5 additions & 0 deletions kronos/cmd/record.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
from oslo_log import log as logging

from kronos.clients.nova import NovaClient
from kronos.clients.placement import PlacementClient
from kronos.clients.prometheus import PrometheusClient
from kronos.common.config import register_opts
from kronos.common.snapshot import write_snapshot
Expand Down Expand Up @@ -72,8 +73,12 @@ def main() -> int:

nova = NovaClient(CONF)
prometheus = PrometheusClient(CONF)
placement = (
PlacementClient(CONF) if CONF.engine.require_cpu_compatibility else None
)
target = write_snapshot(
parent_dir, nova, prometheus, policies, aggregate_names,
placement=placement,
)

LOG.info("Snapshot written to %s", target)
Expand Down
33 changes: 32 additions & 1 deletion kronos/cmd/replay.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,10 @@
from oslo_log import log as logging

from kronos.clients.nova import ComputeService, Instance
from kronos.clients.placement import PlacementClient, ProviderSnapshot
from kronos.clients.prometheus import PrometheusHealth, QueryResult
from kronos.common.config import register_opts
from kronos.common.exceptions import AggregateNotFound
from kronos.common.exceptions import AggregateNotFound, PlacementClientError
from kronos.common.messaging import UNASSIGNED_TOPIC_MARKER
from kronos.engine.cooldown import CooldownTracker
from kronos.engine.loop import EngineLoop
Expand Down Expand Up @@ -295,6 +296,7 @@ def main() -> int:
prometheus=prometheus, # type: ignore[arg-type]
cooldown=cooldown,
timings=timings,
placement=ReplayPlacementClient(snapshot_dir),
)

started = time.perf_counter()
Expand All @@ -307,3 +309,32 @@ def main() -> int:
_log_timings(timings, total)

return 0


class ReplayPlacementClient(PlacementClient):
def __init__(self, snapshot_dir: Path) -> None:
self._traits_path = snapshot_dir / "placement" / "cpu_traits.json"

def fetch_cpu_traits(self, hosts: set[str]) -> dict[str, frozenset[str]]:
if not self._traits_path.exists():
raise PlacementClientError(reason="Snapshot has no CPU traits")
raw = json.loads(self._traits_path.read_text())
if not isinstance(raw, dict):
raise PlacementClientError(reason="Invalid CPU traits snapshot")
traits: dict[str, frozenset[str]] = {}
for host, flags in raw.items():
if not isinstance(flags, list) or not all(
isinstance(flag, str) for flag in flags
):
raise PlacementClientError(reason=f"Invalid CPU traits for {host}")
if host in hosts:
traits[host] = frozenset(
flag for flag in flags if flag.startswith("HW_CPU_")
)
return traits

def fetch_snapshots(self) -> dict[str, ProviderSnapshot]:
raise PlacementClientError(
reason="Placement claims are not recorded. Disable "
"enforce_placement_claims for offline CPU compatibility replay.",
)
9 changes: 9 additions & 0 deletions kronos/common/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -250,6 +250,15 @@
"identical across all hosts in the aggregate."
),
),
cfg.BoolOpt(
"require_cpu_compatibility",
default=False,
help=(
"Require destination Placement HW_CPU_* traits to contain all source "
"traits. Missing data blocks moves. Empty source sets warn and pass. "
"Independent of enforce_placement_claims. Nova still checks migrations."
),
),
]

prometheus_opts: list[cfg.Opt] = [
Expand Down
26 changes: 24 additions & 2 deletions kronos/common/snapshot.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@
from oslo_log import log as logging

from kronos.clients.nova import NovaClient
from kronos.clients.placement import PlacementClient
from kronos.clients.prometheus import PrometheusClient
from kronos.common.messaging import UNASSIGNED_TOPIC_MARKER
from kronos.policies.models import PoliciesConfig
Expand All @@ -43,6 +44,8 @@ def write_snapshot(
prometheus: PrometheusClient,
policies: PoliciesConfig,
aggregate_names: list[str | None],
*,
placement: PlacementClient | None = None,
) -> Path:
"""Write a complete snapshot under ``parent_dir`` and return its path.

Expand Down Expand Up @@ -77,7 +80,9 @@ def write_snapshot(
),
)

_write_nova(nova, aggregate_names, target)
hosts = _write_nova(nova, aggregate_names, target)
if placement is not None:
_write_cpu_traits(placement, hosts, target)
_write_prometheus(prometheus, policies, target)

(target / "cooldowns.json").write_text(
Expand All @@ -98,7 +103,7 @@ def _write_nova(
nova: NovaClient,
aggregate_names: list[str | None],
output_dir: Path,
) -> None:
) -> set[str]:
nova_dir = output_dir / "nova"
nova_dir.mkdir()

Expand Down Expand Up @@ -163,6 +168,8 @@ def _write_nova(
LOG.error("Failed to list compute services", exc_info=True)
(nova_dir / "services.json").write_text("[]")

return all_hosts


def _write_prometheus(
prometheus: PrometheusClient,
Expand Down Expand Up @@ -234,3 +241,18 @@ def _write_prometheus(
policy.name,
exc_info=True,
)


def _write_cpu_traits(
placement: PlacementClient, hosts: set[str], output_dir: Path,
) -> None:
directory = output_dir / "placement"
directory.mkdir()
try:
traits = placement.fetch_cpu_traits(hosts)
except Exception:
LOG.error("Failed to record CPU traits", exc_info=True)
traits = {}
(directory / "cpu_traits.json").write_text(
json.dumps({host: sorted(flags) for host, flags in traits.items()}, indent=2),
)
Loading
Loading