[update-status] Handle svcs in-transition and unrecognized states - #11025
[update-status] Handle svcs in-transition and unrecognized states#11025karencfv wants to merge 4 commits into
Conversation
|
This PR looks a bit big, but it really isn't. Most of it is just API migration code. |
| // https://github.com/oxidecomputer/omicron/issues/10316 | ||
| // | ||
| // `InTransition` (or state with '*' appended as represented | ||
| // in svcs) is excluded because it is a momentary state |
There was a problem hiding this comment.
| // in svcs) is excluded because it is a momentary state | |
| // in svcs) is excluded because it is a momentary state |
| // More detail in | ||
| // https://github.com/oxidecomputer/omicron/issues/10316 | ||
| // | ||
| // `InTransition` (or state with '*' appended as represented |
There was a problem hiding this comment.
This seems okay for what we're trying to do with this consumer, but it doesn't seem great for a general-purpose layer because it doesn't tell you anything about what state it's currently in or what it's going to. It means you lose all information about its state while it's transitioning.
I believe that at the SMF layer (in the database, visible with svcprop), this information is exposed as state and next_state, and the asterisk gets appended if next_state is not NULL:
$ svcprop -p restarter ssh
restarter/logfile astring /var/svc/log/network-ssh:default.log
restarter/contract count 71
restarter/start_pid count 432
restarter/start_method_timestamp time 1780716452.355913000
restarter/start_method_waitstatus integer 0
restarter/auxiliary_state astring dependencies_satisfied
restarter/next_state astring none
restarter/state astring online
restarter/state_timestamp time 1780716452.357390000
We could similarly expose both here. Or we could add the current and next state to the Transitioning variant?
Or maybe it would also be okay to simply ignore the asterisk? On the grounds that if it's offline*, then it is offline, even though it's transitioning. But that seems likely to lead to false positives while things are starting up. I think it matters to our consumer whether something is offline or offline* because the first is a problem and the second isn't.
Or might we also have false positives today if something is offline and not transitioning yet because its dependencies are still being started? In which case we just need to treat this at a higher level as transient. Or report the state_timestamp too, and only consider something broken if its in one of our broken states and its state hasn't changed recently (as a form of hysteresis)?
As I write that, I wonder if we're going to keep playing whack-a-mole with false positives unless we do something like that.
svcshas two additional "states" that are not in the list of official states for SMF service instances. Fromman svcsThis PR adds two additional variants to
SvcState(and related parsing) to account for these additional "states":UnrecognizedandInTransition. Additionally, only theUnrecognizedvariant is added toSvcEnabledNotOnlineState. An unrecognised state should raise an alarm, whereas an "in-transition" state should not.This should probably be merged until the release branch is cut? It's be nice to give this a little bit of time on the dogfood rack.
Fixes: #10997