Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/workflows/check-testnet-manifest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,3 +34,6 @@ jobs:

- name: Validate cardano-testnet manifest golden file against its schema
run: make testnet-manifest-validate

- name: Validate cardano-testnet status golden files against their schema
run: make testnet-status-validate
11 changes: 10 additions & 1 deletion .github/workflows/haskell.yml
Original file line number Diff line number Diff line change
Expand Up @@ -142,7 +142,7 @@ jobs:
run: |
cabal test cardano-testnet cardano-node cardano-node-chairman cardano-submit-api

- name: Validate produced manifest files against schema
- name: Validate produced manifest and status output against schemas
if: matrix.validate-manifest
run: |
# This job has no nix, that is why we pin the version with pipx separately.
Expand All @@ -159,6 +159,10 @@ jobs:
if ! kill -0 "$testnet_pid" 2>/dev/null; then break; fi
sleep 5
done
# Exit code 0: the chain is producing blocks.
status_rc=0
"$(cabal list-bin cardano-testnet)" status --testnet-dir "$dir" --json > "$RUNNER_TEMP/status.json" || status_rc=$?
cat "$RUNNER_TEMP/status.json"
# SIGINT, not SIGTERM: the harness tears its nodes down on SIGINT only
# (see Testnet.Signal.interruptNodesOnSigINT).
kill -INT "$testnet_pid" 2>/dev/null || true
Expand All @@ -169,6 +173,11 @@ jobs:
exit 1
fi
check-jsonschema --schemafile cardano-testnet/schemas/manifest.schema.json "$dir/manifest.json"
if [ "$status_rc" != 0 ]; then
echo "ERROR: cardano-testnet status exited with $status_rc"
exit 1
fi
check-jsonschema --schemafile cardano-testnet/schemas/status.schema.json "$RUNNER_TEMP/status.json"

- name: Tar failed tests workspaces
if: ${{ failure() }}
Expand Down
8 changes: 7 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,12 @@ testnet-manifest-validate: ## Validate manifest golden file (and optionally MANI
cardano-testnet/test/cardano-testnet-golden/files/golden/manifest.json
$(if $(MANIFEST),nix run .#check-jsonschema -- --schemafile cardano-testnet/schemas/manifest.schema.json $(MANIFEST),)

testnet-status-validate: ## Validate status golden files (and optionally STATUS=path) against their schema
nix run .#check-jsonschema -- --schemafile cardano-testnet/schemas/status.schema.json \
cardano-testnet/test/cardano-testnet-golden/files/golden/status.json \
cardano-testnet/test/cardano-testnet-golden/files/golden/status_no_testnet.json
$(if $(STATUS),nix run .#check-jsonschema -- --schemafile cardano-testnet/schemas/status.schema.json $(STATUS),)

###
### Workbench: cluster shells
###
Expand Down Expand Up @@ -84,6 +90,6 @@ cls:
.PHONY: help lint hlint host-hlint haddock-hoogle stylish-haskell cabal-hashes cli node \
trace-documentation trace-schemas-regenerate trace-schemas-overrides-check \
trace-schemas-overrides-coverage trace-schemas-validate \
testnet-manifest-validate \
testnet-manifest-validate testnet-status-validate \
workbench-ci ci ci-report ci-targets \
clean full-clean cls
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
description: Added `status` command that checks the status of a testnet created with
Comment thread
palas marked this conversation as resolved.
`cardano-testnet`. The output can be either in human readable format or JSON (if
the `--json` flag is specified). The library also exports the `Testnet.Status.Types`,
`Testnet.Status.Run` and `Testnet.Status.Parsers` modules, `isProcessAlive` from
`Testnet.Signal`, `testnetEpochSlots` from `Testnet.Types` and `defaultTestnetDir`
from `Testnet.Start.Types`.
kind:
- feature
pr: 6741
project: cardano-testnet
33 changes: 32 additions & 1 deletion cardano-testnet/README.md
Comment thread
palas marked this conversation as resolved.
Original file line number Diff line number Diff line change
Expand Up @@ -53,14 +53,15 @@ export CARDANO_NODE=path to your executable
-->
<!-- checked-help: cardano-testnet --help -->
```text
Usage: cardano-testnet (cardano | create-env | version | help)
Usage: cardano-testnet (cardano | create-env | status | version | help)

Available options:
-h,--help Show this help text

Available commands:
cardano Start a testnet and keep it running until stopped
create-env Create a sandbox for Cardano testnet
status Check the status of a running testnet
version Show cardano-testnet version
help Show cardano-testnet help
```
Expand Down Expand Up @@ -442,6 +443,36 @@ readers never see a partial file. It is written once and never modified.
A pre-existing manifest from an earlier run in the same directory is deleted
before nodes start.

### Status output

`cardano-testnet status --json` prints the health of a testnet as a single
JSON object: the state of the network, the exit code of the command, why
there is no testnet (when there is none), the
freshest tip, and the state and tip of every node, with the error of a node
that did not answer. The exit code is 0 while
the chain is producing blocks, even when some nodes are not healthy. A script
that needs every node to be healthy can check the `status` field:

```bash
[ "$(cardano-testnet status --testnet-dir testnet --json | jq -r '.status')" = running ]
```

The exit code, for the table and for `--json`:

| Exit code | Meaning |
|-----------|-----------------------------------------------------------|
| 0 | `running` or `degraded`: the chain is producing blocks |
| 1 | the command failed, for example on an unreadable manifest |
| 3 | `stalled`: some nodes answer, but none has a fresh tip |
| 4 | `stopped`: no node answers |
| 5 | `no-testnet`: there is no manifest in the directory |

**Stability promise.** Like the manifest, the output has a strict JSON Schema
(`cardano-testnet/schemas/status.schema.json`, draft 2020-12, unknown fields
are an error) that lives in the repository and is the contract. Any change to
the output shape, including new fields, bumps the schema version, so the
output always matches the schema of its own version exactly.

## Supported versions

cardano-testnet is designed to run the cardano-cli and cardano-node that ship
Expand Down
3 changes: 3 additions & 0 deletions cardano-testnet/app/cardano-testnet.hs
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,14 @@ import Cardano.CLI.Environment (getEnvCli)
import qualified Cardano.Crypto.Init as Crypto

import qualified Options.Applicative as Opt
import System.IO (BufferMode (LineBuffering), hSetBuffering, stderr, stdout)

import Parsers.Run (opts, pref, runTestnetCmd)

main :: IO ()
main = do
hSetBuffering stdout LineBuffering
hSetBuffering stderr LineBuffering
Crypto.cryptoInit

envCli <- getEnvCli
Expand Down
6 changes: 6 additions & 0 deletions cardano-testnet/cardano-testnet.cabal
Original file line number Diff line number Diff line change
Expand Up @@ -132,6 +132,9 @@ library
Testnet.Start.Byron
Testnet.Start.Cardano
Testnet.Start.Types
Testnet.Status.Parsers
Testnet.Status.Run
Testnet.Status.Types
Testnet.SubmitApi
Testnet.TestQueryCmds
Testnet.Types
Expand Down Expand Up @@ -171,6 +174,7 @@ test-suite cardano-testnet-golden
other-modules: Cardano.Testnet.Test.Golden.Config
Cardano.Testnet.Test.Golden.Help
Cardano.Testnet.Test.Golden.Manifest
Cardano.Testnet.Test.Golden.Status
Cardano.Testnet.Test.Golden.Util

type: exitcode-stdio-1.0
Expand Down Expand Up @@ -255,6 +259,8 @@ test-suite cardano-testnet-test
Cardano.Testnet.Test.Node.Shutdown
Cardano.Testnet.Test.Parser
Cardano.Testnet.Test.Manifest
Cardano.Testnet.Test.Status
Cardano.Testnet.Test.StatusRules
Cardano.Testnet.Test.MainnetParams
Cardano.Testnet.Test.SanityCheck
Cardano.Testnet.Test.RunTestnet
Expand Down
139 changes: 139 additions & 0 deletions cardano-testnet/schemas/status.schema.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,139 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://github.com/IntersectMBO/cardano-node/cardano-testnet/schemas/status.schema.json",
"title": "cardano-testnet status",
"description": "Output of 'cardano-testnet status --json': the health of a testnet, found by asking every node in its manifest for its chain tip. The schema is strict: it describes schema version 1 exactly, unknown fields are an error, and any change to the output shape bumps schemaVersion.",
"type": "object",
"required": [
"schemaVersion",
"checkedAt",
"status",
"exitCode",
"chainProducingBlocks",
"bestTip",
"nodes",
"noTestnetReason"
],
"additionalProperties": false,
"properties": {
"schemaVersion": {
"type": "integer",
"const": 1,
"description": "Schema version. Any change to the output shape, including new fields or values, bumps it."
},
"checkedAt": {
"type": "string",
"format": "date-time",
"description": "UTC time the check started (ISO-8601)."
},
"status": {
"type": "string",
"enum": ["running", "degraded", "stalled", "stopped", "no-testnet"],
"description": "State of the whole network. \"running\": every node is ok. \"degraded\": some nodes are ok, but not all. \"stalled\": some nodes answer, but none is ok. \"stopped\": no node answers. \"no-testnet\": there is no manifest in the directory."
},
"noTestnetReason": {
"description": "Why there is no testnet, when the status is \"no-testnet\". \"dir-missing\": the directory does not exist. \"not-ready\": it has genesis files but no manifest: the testnet is still starting, or it never became ready. \"not-a-testnet-dir\": it is not the output directory of a testnet. Null for any other status.",
"oneOf": [
{ "type": "null" },
{ "type": "string", "enum": ["dir-missing", "not-ready", "not-a-testnet-dir"] }
]
},
"exitCode": {
"type": "integer",
"enum": [0, 3, 4, 5],
"description": "Exit code of the command: 0 for \"running\" and \"degraded\", 3 for \"stalled\", 4 for \"stopped\", 5 for \"no-testnet\"."
},
"chainProducingBlocks": {
"type": "boolean",
"description": "True when the exit code is 0: the status is \"running\" or \"degraded\"."
},
"bestTip": {
"description": "The newest tip of all nodes: the one with the highest slot. Null when no node answered with a tip.",
"oneOf": [
{ "type": "null" },
{ "$ref": "#/$defs/tip" }
]
},
"nodes": {
"type": "array",
"description": "One entry per node, in the order of the manifest. Empty for \"no-testnet\".",
"items": {
"type": "object",
"required": ["name", "role", "pid", "pidAlive", "state", "tip", "error"],
"additionalProperties": false,
"properties": {
"name": {
"type": "string",
"description": "Node name, as in the manifest."
},
"role": {
"type": "string",
"enum": ["spo", "relay"],
"description": "\"spo\" for stake pool operators, \"relay\" for relay nodes, as in the manifest."
},
"pid": {
"description": "OS process ID from the manifest, or null when the manifest has none.",
"oneOf": [
{ "type": "null" },
{ "type": "integer" }
]
},
"pidAlive": {
"description": "Whether a process with this pid exists, or null when that is unknown: no pid, or Windows.",
"oneOf": [
{ "type": "null" },
{ "type": "boolean" }
]
},
"state": {
"type": "string",
"enum": ["ok", "stalled", "unreachable", "down"],
"description": "\"ok\": it answers, and its tip is fresh. \"stalled\": it answers, but its tip is too old, or it has no blocks yet. \"unreachable\": it does not answer, but its process may be alive. \"down\": it does not answer, and its process is gone."
},
"tip": {
"description": "The node's tip, or null when it did not answer with one.",
"oneOf": [
{ "type": "null" },
{ "$ref": "#/$defs/tip" }
]
},
"error": {
"description": "Why the node did not answer (an error, or no answer in time), or null when it answered.",
"oneOf": [
{ "type": "null" },
{ "type": "string" }
]
}
}
}
}
},
"$defs": {
"tip": {
"type": "object",
"required": ["slot", "blockNo", "hash", "ageSeconds"],
"additionalProperties": false,
"properties": {
"slot": {
"type": "integer",
"minimum": 0,
"description": "Slot number of the tip."
},
"blockNo": {
"type": "integer",
"minimum": 0,
"description": "Block number of the tip."
},
"hash": {
"type": "string",
"pattern": "^[0-9a-f]{64}$",
"description": "Block header hash of the tip, hex-encoded."
},
"ageSeconds": {
"type": "number",
"description": "Seconds from the start of the tip's slot until the node answered."
}
}
}
}
}
8 changes: 7 additions & 1 deletion cardano-testnet/src/Parsers/Run.hs
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,9 @@ import System.Directory (doesDirectoryExist)
import Testnet.Filepath (unTmpAbsPath)
import Testnet.Start.Cardano
import Testnet.Start.Types
import Testnet.Status.Parsers (cmdCheckStatus)
import Testnet.Status.Run (runCheckStatusOptions)
import Testnet.Status.Types (CheckStatusOptions)
import Testnet.Types (TestnetNode (..))

import Parsers.Cardano
Expand Down Expand Up @@ -62,6 +65,7 @@ opts envCli = Opt.info (commands envCli <**> helper) idm
data CardanoTestnetCommands
= StartCardanoTestnet CardanoTestnetCliOptions
| CreateTestnetEnv CardanoTestnetCreateEnvOptions
| CheckStatus CheckStatusOptions
| GetVersion VersionOptions
| Help ParserPrefs (ParserInfo CardanoTestnetCommands) HelpOptions

Expand All @@ -70,6 +74,7 @@ commands envCli =
asum
[ fmap StartCardanoTestnet (subparser cmdCardano)
, fmap CreateTestnetEnv (subparser cmdCreateEnv)
, fmap CheckStatus (subparser cmdCheckStatus)
, fmap GetVersion (subparser cmdVersion)
, fmap (Help pref (opts envCli)) (subparser cmdHelp)
]
Expand All @@ -79,6 +84,7 @@ runTestnetCmd :: CardanoTestnetCommands -> IO ()
runTestnetCmd = \case
StartCardanoTestnet cmdOpts -> runCardanoOptions cmdOpts
CreateTestnetEnv cmdOpts -> createEnvOptions cmdOpts
CheckStatus cmdOpts -> runCheckStatusOptions cmdOpts
GetVersion cmdOpts -> runVersionOptions cmdOpts
Help pPrefs pInfo cmdOpts -> runHelpOptions pPrefs pInfo cmdOpts

Expand All @@ -96,7 +102,7 @@ runCardanoOptions = \case
-- Create the sandbox, then run cardano-testnet.
-- It is not necessary to update timestamps here, because
-- the genesis files will be created with up-to-date stamps already.
let dirName = fromMaybe "testnet" noEnvOutputDir
let dirName = fromMaybe defaultTestnetDir noEnvOutputDir
conf <- mkConfigAbs dirName
runSimpleApp . runResourceT $ do
logInfo $ "Creating environment: " <> display (tempAbsPath conf)
Expand Down
27 changes: 23 additions & 4 deletions cardano-testnet/src/Testnet/Signal.hs
Original file line number Diff line number Diff line change
Expand Up @@ -10,15 +10,17 @@
module Testnet.Signal
( hardKillProcess
, interruptNodesOnSigINT
, isProcessAlive
) where

#ifdef UNIX
import Control.Monad
import System.Posix.Signals (Handler (..), installHandler, raiseSignal, sigINT,
sigKILL, signalProcess)
import System.Process (ProcessHandle, getPid, interruptProcessGroupOf)
import System.IO.Error (isDoesNotExistError, tryIOError)
import System.Posix.Signals (Handler (..), installHandler, nullSignal, raiseSignal,
sigINT, sigKILL, signalProcess)
import System.Process (Pid, ProcessHandle, getPid, interruptProcessGroupOf)
#else
import System.Process (ProcessHandle, terminateProcess)
import System.Process (Pid, ProcessHandle, terminateProcess)
#endif

import Data.List.NonEmpty (NonEmpty)
Expand Down Expand Up @@ -49,3 +51,20 @@ hardKillProcess hProcess = getPid hProcess >>= mapM_ (signalProcess sigKILL)
#else
hardKillProcess = terminateProcess
#endif

-- | Whether a process with the given pid exists: 'Just' the answer on unix,
-- where signal 0 checks this without sending anything, and 'Nothing' on
-- Windows. A dead process that nobody has waited for yet (a zombie) still
-- exists, and so does an unrelated process that reuses the pid.
isProcessAlive :: Pid -> IO (Maybe Bool)
#ifdef UNIX
isProcessAlive pid
| pid <= 0 = pure (Just False) -- 0 and negative pids name process groups, not one process
| otherwise = do
result <- tryIOError $ signalProcess nullSignal pid
pure . Just $ case result of
Left e | isDoesNotExistError e -> False -- no such process
_ -> True -- signalled, or not allowed to: either way it exists
#else
isProcessAlive _ = pure Nothing
#endif
Loading
Loading