[Alexander-Auer]

feat: backend configuration and a closed registry

Part of #18.

Important

Depends on #27. Merge after #27 (PR 2 of 4; the full order is listed in #27).
Until #27 is merged, this diff also shows the earlier PRs. This PR's own change is one commit:
f90cdd4

A consumer can now say which backend it wants as plain data, and get a backend for it:

from intercede import registry

config = {"kind": "batch", "transport": "ssh", "scheduler": "slurm",
          "host": "batch.example.org", "user": "dirac",
          "known_hosts": "batch.example.org ssh-ed25519 AAAA..."}

async with registry.backend(config) as ce:
    await ce.submit(spec, copies)

Configuration: one flat public model

BackendConfig is the only public configuration class. The consumer uses the same field names for every backend, and never needs to know which classes interCEde uses inside:

kind	applies
arc, htcondor-ce	endpoint, auth (token, proxy)
cloud	endpoint, driver, vm_image, vm_flavor, driver_options, vm_networks, vm_ssh_key, vm_max_lifetime_s
batch + local	scheduler
batch + ssh	scheduler, host, user, known_hosts, port, jump_host, connect_timeout_s, auth (key, password or agent)
all but cloud	queue, container_launcher, max_output_bytes, max_output_files, transfer_timeout_s

The three backend families of IC-ADR-001 §4 are covered:

• monolithic: arc, htcondor-ce
• provider: cloud
• composed: batch, a transport combined with a scheduler

How it is checked. Internally, resolve() turns the flat model into the settings of exactly one backend (_backends/config.py), and the model calls it while it is being built. So mistakes are reported immediately, in the consumer's own field names:

host: does not apply to kind='arc'
user: required for kind='batch' with transport='ssh'; known_hosts: required for ...

Rules:

• Unknown fields and typos are refused, not ignored.
• There is nowhere to put a secret, as IC-ADR-003 requires. driver_options additionally refuses secret-looking keys.
• SSH requires known_hosts, because host key verification is not optional.
• An HTCondor-CE always needs a token.
• Limits must be positive.
• A container_launcher must contain an {image} placeholder.

Registry: closed, lazy, checked

• Closed tables. _BACKENDS, _TRANSPORTS and _SCHEDULERS map a name to "module:Class" and live in interCEde itself, as dictionaries rather than entry points, so nothing outside the library can add to them (IC-ADR-001 Rejected Ideas).
• Lazy imports. A class is imported only when a configuration asks for it, and cached. A test shows that import intercede imports no backend.
• Fails fast. Every registered class provides from_settings(...). The registry builds the object and checks it with isinstance against JobBackend, Transport or Scheduler. A missing module, a missing class, a missing from_settings, or an object that breaks its protocol is reported as a RegistryError at resolution, never at the first call.
• Composition. For batch, the transport is built from its own settings and the scheduler from the batch settings, then both are passed to the batch backend. So ssh+slurm and local+slurm share one class.
• registry.available() lists what a configuration can ask for in this version, for tooling.

Errors. ConfigurationError means the configuration is invalid. The new RegistryError means the configuration is valid, but this version does not ship what it asks for, or an entry is broken.

The tables currently contain only the internal fake, which the conformance suite now builds through the registry. Each real backend from issues #19 to #23 adds one line plus a from_settings. Until then, valid real configurations correctly report, for example, backend 'arc' is not available in this version of interCEde.

Differences from issue #18 and the ADR

• No entry points, no plugins and no override precedence. Issue [Feature]: Typed registry over entry points (backends / transports / schedulers) #18 predates the revised IC-ADR-001, which makes the backend set closed. The tables are plain dictionaries instead.
• One flat public configuration model instead of public per-family models. ADR-001 §5 sketches public per-kind models. This follows the flat shape of the ADR-003 examples ({"kind": "batch", "transport": "ssh", "scheduler": "slurm", "host": ...}) and keeps the per-family settings internal.
• Only kinds that are planned. lsf, sge, oar and torque are left out for now. Adding a kind later is non-breaking.

Tests

• tests/unit/test_config.py: every family, required and refused fields, auth values, limits, and the error messages
• tests/unit/test_registry.py: resolution per family, composition, broken entries, caching, available(), lazy imports, and that calls on the returned object reach the registered class
• tests/unit/_registry_dummies.py: stand-in classes the tests register

163 tests pass on Python 3.11 and 3.13, and pre-commit, including strict mypy, passes.

[aldbr]

A few comments, most are minor.
I just have a question around BackendConfig: I would lean towards different public interfaces rather than 1 big interface. But no strong opinion about it, we can discuss

[aldbr]

What about naming it get_backend()?

[Alexander-Auer]

that is probably a bit nicer yes

[aldbr]

Is there any risk of leaking secrets?
I know it shouldn't be the case, but one could potentially misunderstand the API and try to add a secret

Example:

from intercede import registry

config = read_from_file()
# {"kind": "batch", ...,  "password": "toto4"} ⚠️ 

async with registry.backend(config) as ce:
    await ce.submit(spec, copies)

1 validation error for BackendConfig
  password
    Extra inputs are not permitted [type=extra_forbidden, input_value='toto4', input_type=str]

What do you think?
Can you check if there is any easy way of just displaying expected values (declared in BackendConfig and scalar)?

[Alexander-Auer]

When accidentally passing along a secret like this it did indeed leak in the error message. Instead I made it so values don't get shown in errors to make sure a secret does not get shown just in case.

[aldbr]

Can you see whether this raises an error please?

base = BackendConfig(
    kind="batch",
    transport="ssh",
    scheduler="slurm",
    host="login.example.org",
    known_hosts="login.example.org ssh-ed25519 AAAA…"
)
local = base.model_copy(update={"transport": "local"})   # forgot to clear host/known_hosts

registry.backend(local)

[Alexander-Auer]

The copy skipped validation and the build failed with a raw pydantic error, so get_backend now validates the model again to make sure a ConfigurationError gets raised naming host, user and known hosts.

[aldbr]

This has the potential of creating discrepancies if any of these fields is updated.
It would be probably be useful but would be better if it's automatically generated from the code somewhere in the documentation for instance.
See https://github.com/DIRACGrid/diracx/blob/3223011c5b8cc1d0a77636470ac40e76c19e9e36/pixi.toml#L246 in diracx for instance

[Alexander-Auer]

That's true, I replaced them with a reference generated from the models

[aldbr]

Note: ideally, we want unit tests to be focused on the public API to avoid changing the unit tests for a minor change in a private function.

Unit tests on private components can be interesting if something is complex to test though

[aldbr]

These fields are defined twice (here and in the pydantic models): this can introduce discrepancies.
Ideally you get them from the models like:

LIMIT_FIELDS = tuple(Limits.model_fields)
SSH_FIELDS = tuple(name for name in SSHTransportSettings.model_fields if name not in ("kind", "auth"))

[aldbr]

Out of curiosity, did you choose these names yourself or is it based on any guidelines?

[aldbr]

Here resolve() is called twice: 1st by _fits_the_backend calling resolve() and then here. Is that expected?

[aldbr]

Do we need BackendConfig?

Sounds like a potential source of discrepancy here also.

What about:

BackendConfig = Annotated[
    Union[
        Annotated[ARCConfig, Tag("arc")],
        Annotated[CloudConfig, Tag("cloud")],
        Annotated[BatchLocalConfig, Tag("batch+local")],
        Annotated[BatchSSHConfig, Tag("batch+ssh")]
    ],
    Discriminator(lambda v: f"batch+{v.get('transport')}" if v.get("kind") == "batch" else v.get("kind")),
]

Where ...Config would correspond to your ...Settings classes.

The user would get:

• Generated reference docs: one page per backend with exactly its fields (the current vesion would give one page of ~25 optional fields).
• Typed code
• If we ever publish JSON schema, editors could suggest only the fields valid for the chosen kind.

But the flat model is also convenient:

• One interface, and internal classes can be reorganised without breaking the API.
• Load errors read slightly better today: host: does not apply to kind='arc' vs arc.host: Extra inputs are not permitted.

Do you have any opinion?

[Alexander-Auer]

@aldbr I believe all the commants have been accounted for :)

[aldbr]

Just wondering: shouldn't it be FakeConfig? And shouldn't it be located in src/intercede/config.py with the other ones?

[Alexander-Auer]

It's internal on purpose: a consumer can't select the fake, only the test suites build it, so it stays out of intercede.config. But you're right about the name: since the review the public models are passed to backends directly, so it's just the fake's config now. I renamed it to FakeConfig.