feat: inheritance mode, body-field descriptions, keep soft keywords - #5
feat: inheritance mode, body-field descriptions, keep soft keywords#5K1rL3s wants to merge 4 commits into
Conversation
da4b94c to
32d314d
Compare
Three additions, driven by generating a hand-written client's shape:
- `--inheritance` renders `allOf: [{$ref: Base}, ...]` as a real base class
instead of merging the parent's fields into every subtype. A discriminated
base stays a model (its own properties survive) and its mapped subtypes
inherit from it, re-declaring only the discriminator tag. Declarations are
emitted parent-first so a `class Sub(Base)` statement resolves, and model
constructors become keyword-only because a subclass may pin an inherited
field to a default while adding required fields of its own.
- `IRBodyField.description` carries the schema description of a spread request
body field, which was silently dropped. `IRParameter` already had it.
- `sanitize_identifier` no longer suffixes soft keywords: `type` is a legal
attribute name and an extremely common spec field, so `type_` was noise.
`_` stays reserved.
Verified on a real 130-schema spec: both file layouts x all three serializers
generate code that passes `ruff --isolated` and `mypy --strict`.
Review of 566dc7c found the inheritance path silently changing decoded data in several shapes. Fixes, most severe first: - A `oneOf` + `discriminator` union holder is no longer force-built as a class. It declares no properties, so `--inheritance` produced `class Button: pass` and every `list[Button]` payload decoded into it, dropping each variant's fields. It stays a union alias; only bases with their own properties become classes. Verified: pydantic again decodes `CallbackButton(payload=...)` and `isinstance(x, ButtonBase)` still holds. - The pinned discriminator tag is reserved against the subtype's existing field names. A sibling property whose wire name only differs in case (`Type` vs `type`) snake-cases to the same identifier, and the two class attributes collapsed into one -- destroying the tag, with ruff reporting nothing. - A subtype that restates an inherited property only to attach prose, or to relax it to nullable, now inherits it instead of emitting an override. `v: str | None` over the base's `v: str` is an `[assignment]` error under `mypy --strict`, so `--inheritance --check` failed on ordinary specs. Genuine narrowings (a `Literal` tag over a `str`) are kept. - The base class is resolved from the schema, not from the half-built `_declarations` registry. A base whose own body refers back to its subtype (a recursive hierarchy) has no entry yet, so inheritance silently degraded to a field merge based on nothing but graph traversal order. - Keyword-only constructors are limited to the models in a hierarchy. One `allOf` subtype used to flip every model in the package, breaking positional construction for models with no relation to it. - A discriminated base kept as a class emits its mapping as a comment. No serializer resolves a subtype from a base-class annotation on its own, and `IRModel.discriminator` was read by nobody, so that was dropped on the floor. - `IRBodyField.description` (and `IRParameter.description`) now reach the generated code as PEP 258 attribute docstrings. Both were carried through the IR and rendered nowhere. Also: `IRModel.base` -> `base_model`, so it can't be confused with `IREnum.base` (the enum's "str"/"int" value type); soft keywords use an allow-list over `keyword.issoftkeyword` so a future Python's new soft keyword stays guarded; `_ordered_declarations` breaks an inheritance cycle instead of emitting a class before its base; the two discriminator-mapping loops share one helper; `_inherited_ref` is computed once and passed down. Verified: both file layouts x all three serializers generate code that passes `ruff check --isolated` and `mypy --strict` (only the pre-existing `BaseMethod.__init_subclass__` no-untyped-call remains).
32d314d to
53f0765
Compare
Review of
|
| used = NameRegistry() | ||
| for existing in decl.fields: | ||
| used.reserve(existing.name) | ||
| decl.fields.insert( |
There was a problem hiding this comment.
The pinned tag bypasses _is_narrowing, so --inheritance --check fails on the most common discriminator shape.
_drop_unsafe_overrides runs inside _build_object; this insert runs later, from _convert_ref -> _apply_discriminator_tag. Nothing re-checks the field it adds, so Literal[tag] is emitted over whatever the base declared. When the base types the discriminator property as a $ref to an enum (the idiomatic OpenAPI form) that is not a narrowing.
Worse, the two paths fight each other: if the subtype does restate type, _drop_unsafe_overrides correctly drops it, and then this branch puts it right back unchecked.
Reproduced with Button.type: {$ref: ButtonKind} + mapping: {callback: CallbackButton, link: LinkButton}:
@dataclass(kw_only=True)
class Button:
type: ButtonKind
text: str
@dataclass(kw_only=True)
class CallbackButton(Button):
payload: str
type: Literal["callback"] = "callback"models.py:26: error: Incompatible types in assignment (expression has type "Literal['callback']",
base class "Button" defined the type as "ButtonKind") [assignment]
models.py:32: error: ... "Literal['link']" ... "ButtonKind" [assignment]
The same happens in a 3-level hierarchy where the middle class already pinned the tag to its own Literal.
The conservative rule the rest of the module follows applies here too: only pin when _is_narrowing(LiteralType((value,)), inherited_type) holds, otherwise inherit the base's declaration.
| return self.render_alias(decl) | ||
| return self.render_model(decl) | ||
| body = self.render_model(decl) | ||
| if decl.discriminator is not None: |
There was a problem hiding this comment.
Regression in the default (non---inheritance) path.
This fires for any IRModel whose discriminator is set, and in merge mode _flatten_object copies the base's discriminator down into every allOf subtype (discriminator = discriminator or d). So concrete subtypes now carry a header announcing them as tagged-union bases, with a partial, self-referential mapping - the mapping is resolved while the subtype is being built, so _ref_to_name only knows the subtypes converted so far.
No flag needed to hit it; --serializer adaptix on a plain Pet/Dog/Cat spec:
# discriminator: petType (dog=Dog) <- claims Dog is a base that maps "dog" to itself
# subtype resolution is left to the serializer config
@dataclass
class Dog:
...
# discriminator: petType (cat=Cat, dog=Dog) <- and Cat is a base of Dog?
@dataclass
class Cat:
...The root cause is in _flatten_object: a discriminator belongs to the schema that declares it - the same reasoning the PR already applies to the inherited member. Dropping the discriminator or d merge entirely fixes this and is safe: IRModel.discriminator has no other reader, and the inheritance base re-resolves its own in _build_named.
| parent = self._declarations.get(base) | ||
| if not isinstance(parent, IRModel): | ||
| return | ||
| inherited_types = {f.wire_name: f.type for f in parent.fields} |
There was a problem hiding this comment.
wire_name is the wrong key: shadowing happens on the python name.
Type/type and packSize/pack_size are distinct wire names that produce one identifier. The lookup misses, the field is kept, and the subclass attribute shadows an inherited one of an unrelated type. _build_object's field_names registry only dedups within the model's own fields, so nothing else catches it either.
Reproduced (base B has type: string + packSize: integer; subtype A adds Type: integer + pack_size: string):
@dataclass(kw_only=True)
class A(B):
type2: Literal["a"] = "a"
type: int | None = None # wire "Type" -> shadows B.type: str
pack_size: str | None = None # wire "pack_size" -> shadows B.pack_size: intmodels.py:20: error: Incompatible types in assignment (expression has type "int | None",
base class "B" defined the type as "str") [assignment]
models.py:21: error: ... "str | None" ... "int" [assignment]
Dropping is also the wrong remedy here - the subtype's property is genuinely its own, it just needs a non-colliding identifier. Reserving the subtype's field names against the inherited ones (and letting only a same-wire-name re-declaration keep the name) preserves both fields and both aliases.
| already covers the field, so anything that is not a genuine narrowing is | ||
| dropped and simply inherited. | ||
| """ | ||
| parent = self._declarations.get(base) |
There was a problem hiding this comment.
Only one level of the base chain is checked.
parent.fields holds a subclass's own fields only - by construction, that is the whole point of this mode. So in A <- B <- C, A's fields are invisible when C is checked, and an incompatible re-declaration of an A field goes straight through.
Reproduced with A{v: string}, B: allOf[A] + {b}, C: allOf[B] + {v: integer, c}:
@dataclass(kw_only=True)
class C(B):
v: int
c: str | None = Nonemodels.py:20: error: Incompatible types in assignment (expression has type "int",
base class "A" defined the type as "str") [assignment]
The inherited set has to be accumulated by walking base_model to the root (nearest declaration winning, since that is what mypy compares against).
| dropped and simply inherited. | ||
| """ | ||
| parent = self._declarations.get(base) | ||
| if not isinstance(parent, IRModel): |
There was a problem hiding this comment.
The half-built-registry problem _resolve_base_model fixes is reintroduced here.
_resolve_base_model goes to some length to avoid reading self._declarations for a base that has no entry yet, and documents exactly why:
a base whose own body refers back to this subtype (a recursive hierarchy) is still mid-build and has no entry yet, which would silently downgrade the subtype [...] based on nothing but graph traversal order
This method then does that read anyway and returns silently. In the Node / LeafNode shape covered by test_inheritance_recursive_base_still_subclasses, LeafNode is built while Node is mid-flight, so parent is None, no override is pruned, and a widening restatement (v: str | None over v: str) reaches the output and fails mypy --strict.
Since the decision needs every base's final field set anyway (see the grandparent case), the natural place for this pass is after the whole graph is built - build() already computes a base-before-subclass ordering, so walking that list makes the inherited set final and both this and the multi-level gap disappear.
| @@ -515,34 +620,173 @@ def _flatten_object( | |||
| if not isinstance(sub_schema, dict): | |||
| continue | |||
| p, r, a, d = self._flatten_object(sub_schema, sub_base) | |||
There was a problem hiding this comment.
Wasted recursion, and it makes the new cycle guard dead code.
The recursive _flatten_object call happens before the sub is not inherited test, so in inheritance mode the base's entire property set is merged and then thrown away on the next line (only required survives). That is O(depth) redundant work per subtype.
It also has a correctness consequence for the guard added in _ordered_declarations. A base_model cycle can only come from an allOf cycle, and an allOf cycle dies here first:
A: allOf: [{$ref: B}, {properties: {a}}]
B: allOf: [{$ref: A}, {properties: {b}}]
File "ir/builder.py", line 622, in _flatten_object
p, r, a, d = self._flatten_object(sub_schema, sub_base)
[Previous line repeated 980 more times]
RecursionError: maximum recursion depth exceeded
So _ordered_declarations' grey-marking branch, the logger.warning and the assert are unreachable. (The RecursionError itself predates this PR - merge mode crashes identically - but the guard is presented as handling it.)
Worth noting what that branch would do if it were reachable: by the time it runs, _flatten_object has already skipped the base's properties and _drop_unsafe_overrides has already removed the subtype's re-declarations. Clearing base_model at that point emits a class that has lost every inherited field rather than one that fails to import.
| assignable, because a false yes emits code that fails ``mypy --strict`` while a | ||
| false no merely inherits a slightly less precise type. | ||
| """ | ||
| if sub.annotation() == base.annotation(): |
There was a problem hiding this comment.
A restatement with the same annotation is dropped wholesale, but annotation is not all a restatement can carry. default and description ride on the same IRField, and specs restate a property to change exactly those:
Base: {properties: {mode: {type: string, default: "fast"}}}
Sub:
allOf:
- $ref: '#/components/schemas/Base'
- properties: {mode: {type: string, default: "slow", description: "Slower here."}}Sub silently inherits mode = "fast". The description loss is the more awkward one given this same PR adds descriptions to body fields - the prose the spec author attached to the subtype never reaches the output.
Keeping the field when only default/description differ is sound (identical annotation is always a legal override), so the return False could be conditioned on the restatement adding nothing at all.
| lines.append(f"{spec.py_name}: {spec.marker}[Omittable[{spec.inner}]] = Omitted()") | ||
| # PEP 258 attribute docstring: the only place a parameter's / body field's | ||
| # schema prose can land without changing the constructor signature. | ||
| doc = docstring(spec.description, "") |
There was a problem hiding this comment.
docstring(text, indent) uses indent for two things: the prefix it writes, and the wrap width (88 - len(indent)). Passing "" here and letting render_method_class add the four spaces gets the prefix right and the width wrong.
16 | q: Query[str]|
17 | r"""First paragraph that is long enough to need wrapping across several lines in the output| <- 95 cols
18 | file.|
19 | | <- trailing whitespace
20 | Second paragraph with a backslash: curl \ -H "X: y".|
Model field docstrings pass " " and come out right. E501/W291 are outside ruff's default select so --check stays green, but the 88-column target this helper exists to hit is missed. Rendering with the real indent and stripping it back off before render_method_class re-adds it fixes the width; skipping the prefix on empty lines fixes the trailing whitespace.
| if "enum" in schema and "properties" not in schema: | ||
| return False | ||
| disc = schema.get("discriminator") | ||
| if isinstance(disc, dict) and isinstance(disc.get("mapping"), dict): |
There was a problem hiding this comment.
The docstring is candid that this mirrors _build_named ("kept next to nothing else so the two stay reviewable side by side"), but the mirror is already off by one branch.
_build_named, for a discriminated base that is not _is_object, falls through to _build_discriminated_base, which ends with:
if not members:
self._declarations[name] = self._build_object(name, schema, base_uri)
returnSo an empty or fully unresolvable mapping yields an IRModel, while this predicate returns False for it - a subtype pointing at that base silently downgrades to a merge.
The drift is minor today, but the shape is the concern: two hand-synchronised copies of a dispatch, in a mode where a wrong answer changes the class hierarchy. Having _build_named record its decision (or splitting the classification out and calling it from both) removes the class of bug rather than this instance.
| hierarchy.add(model.base_model) | ||
| self._kw_only_models = frozenset(hierarchy) | ||
|
|
||
| def is_kw_only(self, model: IRModel) -> bool: |
There was a problem hiding this comment.
_kw_only_models starts empty, so is_kw_only answers False for every model until bind_document has run. Both production paths happen to bind (render_models_module does it internally, _write_per_object_layout does it explicitly), but render_declaration / render_model are public and neither documents the requirement.
Get it wrong and the failure is not a missing keyword - it is @dataclass on a subclass whose base ends in a defaulted field while the subclass declares a required one:
TypeError: non-default argument 'payload' follows default argument
at import time of the generated package.
The property is a fact about the IR ("this model is in a hierarchy"), not about a rendering session; computing it from model.base_model plus the document once in the builder, or storing it on IRModel, would make the ordering unrepresentable.
|
|
||
| def render_model(self, model: IRModel) -> str: | ||
| lines = [f"class {model.name}(BaseModel):"] | ||
| lines = [f"class {model.name}({model.base_model or 'BaseModel'}):"] |
There was a problem hiding this comment.
Pydantic merges model_config down the MRO, so on a subclass this line is a no-op:
class CallbackButton(Button):
model_config = ConfigDict(populate_by_name=True) # already true via Button
type: Literal["callback"] = "callback"It also reads as if the subclass were deliberately overriding the parent's config. Guarding on model.base_model is None keeps the emitted hierarchy honest.
|
|
||
| @staticmethod | ||
| def _discriminator_comment(disc: Discriminator) -> str: | ||
| mapping = ", ".join(f"{value}={name}" for value, name in sorted(disc.mapping.items())) |
There was a problem hiding this comment.
_discriminator_comment emits
# discriminator: type (callback=CallbackButton, link=LinkButton)
# subtype resolution is left to the serializer config
and render_alias, just below, emits
# discriminator: type (tagged-union wiring is left to the serializer config)
Same information, same audience, two wordings and two layouts - and the alias form drops the value->class mapping even though IRAlias.discriminator.mapping carries it, which is the part a reader actually needs to wire tagged decoding. Routing render_alias through the new helper gives both forms the mapping and leaves one string to maintain.
| declared = self._declarations.get(base_type.name) | ||
| if declared is not None: | ||
| return base_type.name if isinstance(declared, IRModel) else None | ||
| resolved = self._resolver.resolve_ref(inherited["$ref"], base_uri) |
There was a problem hiding this comment.
_convert_ref on the line above already resolved this $ref (it needs the pointer to key _ref_to_name); this resolves it again for the same pointer. RefResolver.resolve_ref has no memoisation and re-splits/re-walks the document each call, and this now runs for every allOf subtype in the spec.
Only the key and resolved.value are needed here, both of which _convert_ref had in hand - returning them (or looking the pointer up from _ref_to_name) drops one full resolution per subtype.
There was a problem hiding this comment.
Correction to the scope claim above: the extra resolve_ref is guarded by if declared is not None: return ..., so it only runs when the base has no _declarations entry yet - the recursive-hierarchy case, not every allOf subtype. The duplicate resolution is real but rare, so I left it alone.
Follow-up review of the inheritance mode found four shapes where the generated
package failed `mypy --strict`, so `--inheritance --check` broke on ordinary
specs, plus one regression in the default (merge) path.
Override pruning moves out of `_build_object` into `_reconcile_inheritance`, a
whole-graph pass over the already base-before-subclass ordering. What a subtype
may keep depends on the final field set of every class above it, which is not
knowable while the subtype is being converted. That single move fixes:
- The pinned discriminator tag is now checked like any other override. It is
added after the model is built, so nothing used to look at it, and a base that
types the discriminator property as a `$ref` to an enum -- the idiomatic
OpenAPI form -- got `type: Literal['callback']` over `type: ButtonKind`. The
two paths also fought: a subtype restating the tag had it dropped as unsound
and then put straight back unchecked.
- The full base chain is consulted, not just the direct parent. A subclass
carries only its own fields, so in `A <- B <- C` a re-declaration of an `A`
field was invisible from `B`.
- A base that is still mid-build no longer silently skips the check. That is the
recursive-hierarchy case `_resolve_base_model` was rewritten for; reading the
half-built registry here reintroduced the same order dependence.
Two more override bugs, fixed alongside:
- Inherited fields were keyed by wire name, but shadowing happens on the python
name: `packSize` on the base and `pack_size` on the subtype are different
properties that collapse onto one attribute. Neither may be dropped, so the
subtype's is renamed and aliased back instead of shadowing the inherited one.
- A field re-declaring an inherited wire name now lands on the inherited
attribute. Otherwise the class has two attributes for one wire key, which
adaptix rejects outright ("fields point to the same path").
- A restatement that changes only the `default` is kept. An identical annotation
is always a legal override, and dropping it handed the subtype the base's
value.
Default-mode regression: `_flatten_object` copied a base's discriminator down
through `allOf`, and the new renderer turns any model carrying one into a
`# discriminator:` header. Every concrete subtype was announced as a
tagged-union base, with a mapping resolved only as far as the graph walk had
got (`# discriminator: petType (dog=Dog)` above `class Dog`). A discriminator
describes the schema that declares it, so it is no longer merged in at all --
`IRModel.discriminator` has no other reader, and an inheritance base re-resolves
its own in `_build_named`.
Also: parameter/body-field docstrings are rendered at the indent they actually
sit at, so wrapping targets 88 columns instead of 92 and blank paragraph
separators no longer carry trailing whitespace; `render_alias` reuses
`_discriminator_comment`, which also gives the alias form the value -> class
mapping it was dropping; pydantic subclasses no longer repeat the inherited
`model_config`.
Verified: 7 specs x 3 serializers x 2 layouts generate packages that pass
`mypy --strict` (only the pre-existing `BaseMethod.__init_subclass__`
no-untyped-call remains), adaptix builds a dumper for every generated model, and
all three serializers round-trip the collision cases.
|
Pushed Fixed
One extra bug surfaced while fixing #3: renaming the sibling freed the name but left the tag on Skipped
Verification: 7 specs × 3 serializers × 2 layouts. Every package passes |
…ursing Review of 566dc7c..2847eee found the mode itself sound -- seven adversarial specs x 3 serializers x 2 layouts generate packages that pass `ruff` and `mypy --strict` and round-trip every wire key -- but two gaps worth closing before merge. `--inheritance` had no compile gate. Both follow-up commits fixed generated code that failed `mypy --strict`, which the unit tests cannot see: they assert on the IR, and a subclass declaration is exactly where a slightly-wrong IR turns into an `[assignment]` error. The gate now runs the hierarchy spec through all three serializers x both layouts. Per-object is not redundant: the base class is the one model reference that has to be imported at runtime rather than deferred into the `TYPE_CHECKING` block. Alongside it, a round-trip test in `test_behavior.py` for what ruff and mypy cannot catch: two attributes collapsing onto one identifier silently drops a value, and two attributes left pointing at one wire key makes adaptix refuse to build the retort. Both were real bugs in 53f0765/2847eee. The shared `hierarchy_spec` fixture carries one schema per rule -- enum-typed tag, prose-only restatement, nullable relaxation, default-only override, `packSize`/`pack_size` and `type`/`Type` collisions, a three-level chain, and a base whose body refers back to its subtype. The cycle fix: `_ordered_declarations` breaks an inheritance cycle by dropping the base edge, but nothing could ever reach it. An inheritance cycle needs an `allOf` cycle, and `_flatten_object` followed one until the interpreter's stack gave out -- a pre-existing crash in both modes, now a skipped member and a warning. Output for every acyclic spec is byte-identical.
Review of 566dc7c..2847eee — ready to merge, with one commit addedVerdict: the inheritance mode is correct. I could not break it. What I did add is the safety net the two follow-up commits argue for. What I verified independentlyBuilt two adversarial specs (beyond the ones in the tests) and generated 3 serializers x 2 layouts from each:
Every package: Gaps closed (712779d)1. 2. No round-trip test. ruff and mypy are both blind to the failure that 53f0765 fixed: two attributes collapsing onto one identifier silently drops a value, and two attributes on one wire key makes adaptix refuse to build the retort. Added to 3. Known limitations, deliberate and documented
|
Three additions, driven by generating a hand-written client's shape:
--inheritancerendersallOf: [{$ref: Base}, ...]as a real base class instead of merging the parent's fields into every subtype. A discriminated base stays a model (its own properties survive) and its mapped subtypes inherit from it, re-declaring only the discriminator tag. Declarations are emitted parent-first so aclass Sub(Base)statement resolves, and model constructors become keyword-only because a subclass may pin an inherited field to a default while adding required fields of its own.IRBodyField.descriptioncarries the schema description of a spread request body field, which was silently dropped.IRParameteralready had it.sanitize_identifierno longer suffixes soft keywords:typeis a legal attribute name and an extremely common spec field, sotype_was noise._stays reserved.Verified on a real 130-schema spec: both file layouts x all three serializers generate code that passes
ruff --isolatedandmypy --strict.Description
Please include a summary of the change and specify which issue is being addressed. Additionally, provide relevant motivation and context.
Fixes # (issue number)
Type of change
Please delete options that are not relevant.
Checklist
uv run ruff checkanduv run ruff format --checkshow no errors)uv run mypy