Tufaceous v2: project rewrite / RFD 621 implementation#45
Open
Tufaceous v2: project rewrite / RFD 621 implementation#45
Conversation
the primary purpose of this change is `ArtifactsExt::fake`, which is guaranteed to return the same set of `Artifacts` that you would get from generating a repository, but without requiring async. this also replaced the struct variants of `KnownArtifactTags` with named structs (RFD 643).
only used within wicket, easy to move there
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I don't think this is 100% to where I want it to be yet but I have to cut a PR at some point.
This is essentially a complete redo of Tufaceous. I have been developing it in parallel with an Omicron changeset that integrates it (which you can see the WIP tree here: https://github.com/oxidecomputer/omicron/tree/iliana/rfd621).
Here are the main ideas:
All-in-one API
Everything a consumer needs for creating and reading repositories is in Tufaceous; there's no need to directly use APIs from tough. (The one exception to this currently is using non-ed25519 signing keys; when I get more of the signing tooling done I will probably add this.)
Artifact tags
RFD 621 goes into detail here, but the name-version-kind triple really doesn't do a good job of differentiating artifacts in practice. This system was developed in order to future-proof Nexus against future kinds of artifacts, but when we started this system we didn't fully understand what types of artifacts we need in the repo.
All artifacts that can go in the repo are described with
KnownArtifactTags, an enum type that serializes to a mapping of string keys to string values. It is not possible to use the library to create a repository with "unknown" tags. When an older Nexus, using an older Tufaceous v2 library, interacts with a repo from the future with unknown tags, it will still store them in the database but ignore them for planning purposes.ArtifactSetThis is basically a
BTreeSet<Artifact>which has an index based on an artifact's known tags. This structure replaces most of the code inupdate-commoncurrently in Omicron (apart from reading ZIP archives, below).Builder API for creating / editing archives
In v1, repos were generated from manifests, but you couldn't automatically generate a manifest from a repo. This meant that patching a repo to add/change artifacts was extremely difficult, and it was usually easier to just use our releng tooling.
Instead we use a builder API to create and edit archives using the
RepositoryEditor. You can start with an empty repository, or a fake repository with one of every artifact in it, or an already-loaded repository in order to modify it.A significant amount of the code (and related abstractions) in the
editmodule is used for guessing artifact tags. This was done to make the CLI tooling bearable. I've added a note to the new README that indicates that future artifacts should be "guessable" by this tooling.ZIP archive implementation
Tufaceous now has a ZIP transport implementation for tough, so Omicron no longer needs to unpack ZIP archives to disk in order to load the repository. There is a lengthy comment at the start of
lib/src/zip_transport.rson some of the pitfalls of using ZIP archives and the mitigations we've taken.v1 compatibility
The v2 repository format is not compatible with the v1 format.
RepositoryLoadercan read v1 repos, unpacking composite artifacts along the way. This library is not intended for creating v1 repos, and we expect therelengtool will carry a dependency on both Tufaceous v1 and v2 until we stop producing v1 repos. After that happens, we can start to remove v1 compatibility code from v2.The
tufaceous verifysubcommand has v1 compatibility enabled and can be used to check for problems in the conversion (in fact, writing the problem checker and using it on our v1 repos made me find problems in the compatibility code!).Currently this code cannot import a v1 repository into a [
RepositoryEditor], and thus cannot directly convert a v1 archive into a v2 archive. I would like to add this functionality at some point (there is some additional complexity around handling the composite artifacts).