Skip to content

[update-status] Handle svcs in-transition and unrecognized states - #11025

Open
karencfv wants to merge 4 commits into
oxidecomputer:mainfrom
karencfv:handle-intermediate-states
Open

[update-status] Handle svcs in-transition and unrecognized states#11025
karencfv wants to merge 4 commits into
oxidecomputer:mainfrom
karencfv:handle-intermediate-states

Conversation

@karencfv

@karencfv karencfv commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

svcs has two additional "states" that are not in the list of official states for SMF service instances. From man svcs

Absent or unrecognized states are denoted by a question mark
(?) character. An asterisk (*) is appended for instances in
transition, unless the NSTA or NSTATE column is also being
displayed.

This PR adds two additional variants to SvcState (and related parsing) to account for these additional "states": Unrecognized and InTransition. Additionally, only the Unrecognized variant is added to SvcEnabledNotOnlineState. An unrecognised state should raise an alarm, whereas an "in-transition" state should not.

This should probably be merged until the release branch is cut? It's be nice to give this a little bit of time on the dogfood rack.

Fixes: #10997

@karencfv

karencfv commented Aug 7, 2026

Copy link
Copy Markdown
Contributor Author

This PR looks a bit big, but it really isn't. Most of it is just API migration code.

@karencfv
karencfv marked this pull request as ready for review August 7, 2026 00:53
Comment thread illumos-utils/src/svcs.rs
// https://github.com/oxidecomputer/omicron/issues/10316
//
// `InTransition` (or state with '*' appended as represented
// in svcs) is excluded because it is a momentary state

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// in svcs) is excluded because it is a momentary state
// in svcs) is excluded because it is a momentary state

Comment thread illumos-utils/src/svcs.rs
// More detail in
// https://github.com/oxidecomputer/omicron/issues/10316
//
// `InTransition` (or state with '*' appended as represented

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems okay for what we're trying to do with this consumer, but it doesn't seem great for a general-purpose layer because it doesn't tell you anything about what state it's currently in or what it's going to. It means you lose all information about its state while it's transitioning.

I believe that at the SMF layer (in the database, visible with svcprop), this information is exposed as state and next_state, and the asterisk gets appended if next_state is not NULL:

$ svcprop -p restarter ssh
restarter/logfile astring /var/svc/log/network-ssh:default.log
restarter/contract count 71
restarter/start_pid count 432
restarter/start_method_timestamp time 1780716452.355913000
restarter/start_method_waitstatus integer 0
restarter/auxiliary_state astring dependencies_satisfied
restarter/next_state astring none
restarter/state astring online
restarter/state_timestamp time 1780716452.357390000

We could similarly expose both here. Or we could add the current and next state to the Transitioning variant?

Or maybe it would also be okay to simply ignore the asterisk? On the grounds that if it's offline*, then it is offline, even though it's transitioning. But that seems likely to lead to false positives while things are starting up. I think it matters to our consumer whether something is offline or offline* because the first is a problem and the second isn't.

Or might we also have false positives today if something is offline and not transitioning yet because its dependencies are still being started? In which case we just need to treat this at a higher level as transient. Or report the state_timestamp too, and only consider something broken if its in one of our broken states and its state hasn't changed recently (as a form of hysteresis)?

As I write that, I wonder if we're going to keep playing whack-a-mole with false positives unless we do something like that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fix SMF service parsing for intermediate states

2 participants