Ducktape remote start - #13432
Open
maksaska wants to merge 9 commits into
Open
Conversation
Until now ducktests could be run either locally in Docker via
tests/docker/run_tests.sh, or on a real VM cluster from Jenkins by
hand-assembling a long `ducktape ...` command line with a large inline
--globals JSON blob. There was no supported way to launch a run against
the VM cluster from an engineer's machine. This fills that gap.
New package modules/ducktests/tests/ducktests_remote/, exposed as the
`ducktests-remote` console script:
run compose the artifacts, launch ducktape detached, follow the log
status state of a run, or a table of every run (--all)
logs print or follow a run's ducktape.log
stop terminate a run and clean the workers behind it
fetch download the reports to the coordinator
doctor check the coordinator, the runner and every worker
provision bring the workers to the state the Docker image guarantees
deploy copy distributions to <install_root>/<name> on every worker
clean kill stale Ignite JVMs and remove work directories
keys install the runner identity and authorise it on the workers
Design points worth calling out:
* Every command goes through a Transport (local, ssh, or proxied through
a third host), so --runner local and --runner some-host take identical
code paths above the transport boundary. Nothing shells out to ssh
outside a transport implementation.
* All run state lives on the runner, never on the coordinator, so any
coordinator can inspect, follow or stop a run that a different one
started.
* The package never imports ducktape. It drives ducktape on the runner
and must stay installable on a coordinator that has none; a unit check
fails if that ever changes.
* Runs are detached from second zero via setsid + a pid/exit_code
wrapper. ducktape's runner is an ordinary foreground process, so a
dropped SSH session would otherwise SIGHUP it mid-run and leave Ignite
JVMs alive on every worker. Ctrl-C during --follow detaches; stopping
takes a deliberate second interrupt.
* The Jenkins --globals blob is replaced by layered YAML profiles with
${env:}/${file:} interpolation. A missing variable is a hard error
naming the variable and the file. Resolved secrets are registered with
a value-based redactor and masked in everything the CLI prints;
globals.json is written mode 0600 and always excluded from fetch.
--globals-json/--globals-file keep the existing blob working verbatim
as the migration path.
* No cluster lease: this deployment has a single runner, and queueing
belongs to whatever schedules the runs. Exit code 3 is reserved.
Verified against the sources rather than assumed:
* ducktape 0.13 accepts a file path for --globals (command_line/main.py
checks os.path.isfile first), so the composed blob is referenced by
path and never crosses a shell command line.
* The cluster file schema is the one RemoteAccountSSHConfig accepts.
* ignitetest resolves a distribution home as <install_root>/<product>
where product is str(IgniteVersion(v)), which normalises: ise--6 maps
to /opt/ise-6. Because a fork can override product, doctor reports a
missing directory as a WARN listing what it did find.
* sudo is needed only for `sudo iptables` from
IgniteAwareService.drop_network, reached by exactly two suites:
discovery_test.py and cellular_affinity_test.py. Everything else runs
unprivileged, and doctor says so.
* ducktape's loader puts the test tree on sys.path itself
(loader.py::_add_top_level_dirs_to_sys_path), so synced sources do not
need installing; only the pinned requirements have to be in the venv.
* provision's package list is derived from docker/Dockerfile, with a
comment naming it as the source of truth.
Adds PyYAML to docker/requirements.txt: config and profile parsing is
the package's only dependency beyond the standard library.
Unit checks live in ducktests_remote/checks/ as check_*.py with Check
classes, matching what [pytest] in tox.ini collects. 164 checks, no
network, no Docker, no ducktape. flake8 clean.
A runner inside a corporate network cannot reach PyPI, and a worker rarely
carries the JDK the tests need on its non-interactive PATH. Both were
assumptions the CLI made silently.
pip: new `pip` config section (index_url, extra_index_url, trusted_host,
timeout, retries, cert) rendered into command-line flags by pipconf and
applied to every pip command the CLI runs. `run --install-sources` reached
PyPI regardless of the configured index before this; it no longer does.
`provision.pip_index_url` moves to `pip.index_url`.
java: new `java` config section and a resolution ladder in java.py, shared by
`provision --only jdk`, `provision --only ssh-env` and `doctor` - an explicit
java.home, else the JVM already on PATH when its major matches, else a JDK
under java.search_paths, else java.archive delivered from the coordinator
reusing deploy's staging and atomic swap. The selected JDK is written to both
~/.ssh/environment and a marked block above the interactivity guard in
~/.bashrc, then verified over a fresh non-interactive session, because that
is the JVM ignite.sh, the Kafka service, jmxterm and jvm_utils.java_version
actually get. doctor fails preflight on a major that does not match.
`provision.jdk_major` moves to `java.major`.
${env:} and ${file:} placeholders are now resolved in every config section,
not only in globals; examples/cluster.yaml has always advertised this, but
the values were passed through literally.
The README is a tour: install, describe a cluster, get a run going. It cannot
carry the class of question that actually comes up once the tool is in use -
"when exactly is the JDK copied to the workers", "which layer set this value",
"what does this command do, in what order, and what crosses the network".
Seven documents under ducktests_remote/docs/, with index.md routing a question
to the one that answers it:
concepts the three roles, what lives where, the invariants
configuration every key, its default and who reads it; layering, ${env:},
redaction
commands every command and flag, each with its order of operations and
a table of what it transfers, to whom, and when
java why PATH decides rather than JAVA_HOME, the resolution
ladder, and the exact conditions under which an archive is
delivered
runs the run directory, run states, detach/follow/stop, exit codes
troubleshooting symptom to cause to fix, plus the ssh failure classes
internals module map, transport contract, fan-out, and the ignitetest
facts this CLI is pinned to
Documentation only; no behaviour change.
Test paths now mean what they mean in the Docker flow. ducktape is started in <checkout>/modules/ducktests/tests, the working directory docker/run_tests.sh uses, so a run reads ducktests-remote run ./ignitetest/tests/smoke_test.py::Cls.test instead of repeating modules/ducktests/tests in every path. Paths are accepted in whatever form resolves on the coordinator - relative to the current directory, to the tests directory, or to the checkout root - and are rewritten from there, so the previous form keeps working. A path inside the checkout but outside the tests directory becomes a runner-side absolute path, which is the only form that cannot be misread once the working directory has moved. The checkout itself is found by walking up from the current directory, and a source root that is not one is refused before anything is uploaded. Standing in the wrong place used to sync that directory to the runner and surface minutes later as MISSING_REQUIREMENTS, blaming the pip index for a file that was never there; that exit is now reported separately from a pip failure. java.archive no longer assumes how deeply the JDK is wrapped. bin/java is located in the member list and everything above it stripped, so a hand-repacked tarball (openjdk-17/jdk-17.0.11+9/bin/java) and one with stray entries beside the JDK are both accepted, where before only depth 0 and depth 1 were. The shallowest bin/java wins, so a bundled JRE cannot drag the strip depth with it. A macOS build is still refused, now by checking the resolved prefix rather than by guessing in the error text. Two defects found while reading that code: * .tar.xz and .tar.bz2 passed the coordinator-side check and then failed on every worker, because the extract was a hardcoded `tar -xzf`. The flag now follows the suffix. * An unpacked java.archive directory was packed inside the per-host operation, so twelve workers meant twelve concurrent gzips of the same JDK on the coordinator. JdkPayload packs it once, caches the manifest with it, and cleans up after the fan-out.
ignite-dev is normally a link to a source checkout, and a built checkout is well over a gigabyte of which the workers open almost nothing: they need modules/*/target/*.jar, modules/*/target/libs/*.jar, bin/ and the ducktests certs. deploy had no way to say that, so the whole tree went over the wire to every host. Add --exclude, a .ducktests-deploy.ignore file at the root of a distribution, and deploy.exclude in the configuration, in that order of precedence. The patterns are the rsync-style ones the source sync already uses. Excludes default to nothing, so a release directory is still shipped byte for byte. The manifest is built from the same filtered list as the tarball; otherwise a host would be reported up to date while holding a different set of files. The ignore file is deliberately not called .ducktestsignore: when ignite-dev links to a checkout the distribution root and the source root are the same directory, and the two lists are opposites - the source sync drops target, and deploy keeps little else.
A distribution is all-or-nothing by design: rebuild one module and the manifest hash for the whole tree changes, so every host is redeployed. Until now that meant re-tarring and re-uploading everything, which for a linked checkout is a gigabyte per host to deliver one jar. Fill the staging directory with rsync instead, hardlinked with --link-dest against the deployment already on the host: unchanged files are never sent and cost no disk, and the tree is still built from scratch and still swapped in atomically, so an interrupted transfer cannot leave a live distribution half updated. Deleting the old tree afterwards only drops link counts. rsync is fed the exact file list on stdin rather than its own --exclude patterns, because its matching rules differ from is_excluded's and a distribution that differs from the manifest describing it is the one bug the manifest exists to prevent. The tarball path stays for --no-rsync, deploy.rsync: false, --via, a worker or coordinator without rsync, and a Windows coordinator, where rsync would read C:/dist/ignite-dev as a host named C. rsync is probed per host, so a mixed cluster falls back only where it must, and the tarball is now built lazily - when every host takes the fast path, nothing is compressed at all.
Sending 3.5 GB to twelve machines is a normal deploy and takes minutes. A terminal that prints nothing for that long is indistinguishable from one that has hung, and the operator cannot tell a slow host from a stuck one. Add progress.py: a display written to from the fan-out threads and drawn by a daemon thread of its own, so a slow terminal cannot slow a transfer and an update never blocks on a redraw. On a terminal it redraws a block, one row per host plus a total; in a log, under --verbose, or wherever stderr is not a terminal it prints one aggregate line every 15 seconds. --quiet, --dry-run and --no-progress swap in a NullProgress with the same surface, so no caller has to ask whether the display is on. The total is the mean of the per-host fractions rather than bytes against a predicted total: rsync sends only what differs, so a denominator of distribution size times host count would stall at 12% and finish there. Byte counts come from --info=progress2 on the rsync path. On the tarball path they come from a new upload_watched, which streams the file into ssh 'cat > path' and counts the chunks, because scp's own meter is written for a terminal and suppressed whenever its output is a pipe - which it always is here. scp stays the default for every unwatched upload. rsync now takes its file list as a file rather than on stdin: the output side is read as it arrives, and one thread pumping both pipes of a process deadlocks as soon as either fills.
Both move hundreds of megabytes and both were silent: the source sync is a single 170 MB upload to the runner before a run starts, and `provision --only jdk` sends a JDK to every host that lacks one. Lift build_progress out of deploy into progress.py, so one function decides for all three commands, and give upload_dir an on_progress callback: rsync grows --info=progress2 and is read as it arrives, and the tar fallback goes through upload_watched. parse_rsync_progress moves to transport.py next to the rsync invocations that produce it. Reporting is not free - rsync grows a meter, and a watched upload leaves scp for a streamed `cat` - so callers gate the wiring on progress.watching rather than passing a callback that a NullProgress would throw away. --no-progress and --quiet now really do restore the plain transfer. The JDK step reports only when there is an archive to deliver; probing for a JDK already on the host is one round trip and needs no display.
A watched upload feeds the payload into `ssh cat > path` itself, because scp
hides its meter when stdout is a pipe. It closed stdin to signal end of file
and then called communicate(), which flushes stdin before closing it and guards
only BrokenPipeError - so flushing the handle we had just closed raised
ValueError('flush of closed file'). Every host failed at exactly 100%, with the
payload already on the far end and a staging directory left holding it.
Do the bookkeeping here instead: drain both output pipes on threads for the
whole transfer, and close stdin once. The draining is not incidental. Nothing
read those pipes while a gigabyte was being written, so an ssh that says
anything on the way in - a login banner, a host-key notice - filled its pipe
and stopped the transfer for good.
Staging directories are now discarded on any exit that is not a successful
swap, in deploy and in provision's jdk step, and prepare_script sweeps what an
earlier attempt left. Each holds a whole copy of the distribution under a
dot-prefixed name with a random suffix, so nothing would ever reclaim them.
The only checks of upload_watched went through LocalTransport, which never
reaches this code. The new ones run a real child process.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Thank you for submitting the pull request to the Apache Ignite.
In order to streamline the review of the contribution
we ask you to ensure the following steps have been taken:
The Contribution Checklist
The description explains WHAT and WHY was made instead of HOW.
The following pattern must be used:
IGNITE-XXXX Change summarywhereXXXX- number of JIRA issue.(see the Maintainers list)
the
green visaattached to the JIRA ticket (see tabPR Checkat TC.Bot - Instance 1 or TC.Bot - Instance 2)Notes
If you need any help, please email dev@ignite.apache.org or ask anу advice on http://asf.slack.com #ignite channel.