Repository navigation
Expand file tree
/
Copy pathcli.py
More file actions
2814 lines (2505 loc) · 121 KB
/
Copy pathcli.py
File metadata and controls
2814 lines (2505 loc) · 121 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
"""The ``metaobjects`` console-script — codegen `gen` + drift `verify` (SP-E Unit 2).
Two subcommands:
metaobjects gen <metadataDir> --out <dir> [--package <pkg>]
[--templates <dir>] [--template-spec <json>]
Load metadata, run the Python codegen generator suite, and write files
under ``--out`` (guarded by the @generated header). Prints each written
file. Non-zero exit on a load error. ``--template-spec`` appends the
declarative Mustache template generators (scope perEntity/perPackage/
perModel + outputPattern) described by a JSON spec.
``--templates <dir>`` is the on-disk root every template ref resolves
under. It feeds BOTH the ``--template-spec`` pass and the ``render-helper``
generator's build-time drift gate; until 1.0.5 it reached only the former,
so ``--generators render-helper`` read a hardcoded ``templates`` whatever
this flag said.
metaobjects verify <metadataDir> [--codegen] [--templates] [--db URL] ...
Drift gate with explicit subverbs (ADR-0021 D2 — one verify vocabulary
across ports):
--codegen regenerate into a temp dir + diff against the committed
``--out`` tree (Python's historical default behavior).
``--column-naming`` must match the strategy ``gen`` was
run with (R31) or the ``names`` generator's constants
report spurious drift.
--templates template/prompt ``{{field}}`` ↔ payload-VO drift — the
render ``verify()`` gate — resolving each ``template.*``
node's refs via a filesystem provider rooted at
``--templates-root``.
--db URL REJECTED in the Python port (exit 2): schema verify is the
migrate engine (ADR-0015), not this CLI.
Combining flags runs each requested mode and aggregates the exit code
(non-zero if ANY mode drifts). A bare ``verify`` keeps the historical
default = ``--codegen`` (back-compat) and prints a one-line note that the
explicit subverbs exist.
Named ``metaobjects``, NOT ``meta``: ``meta`` is the Node schema CLI. This CLI
intentionally has NO ``migrate`` subcommand — schema is owned by the Node
toolchain (ADR-0015). The ``--codegen`` mode shares the exact same generation
code path as ``gen`` (verify = gen-to-temp + diff), so drift can never be a
generator-wiring divergence between the two commands.
"""
from __future__ import annotations
import argparse
import dataclasses
import json
import os
import re
import sys
import tempfile
from pathlib import Path
from typing import Callable
from metaobjects import MetaDataLoader
from metaobjects.errors import ErrorCode, MetaError, ParseError
from metaobjects.config.dependencies import (
Collection,
imported_from,
refuse_library_package_misuse,
refuse_unowned_packages,
)
from metaobjects.config.neutral_config import read_neutral_config
from metaobjects.field_lint import FieldLintFinding, lint_duplicate_fields, lint_reference_fields
from metaobjects.loader.meta_data_loader import LoadResult
from metaobjects.loader.sources import FileSource
from metaobjects.meta.core.object.meta_object import MetaObject
from metaobjects.agent_context import (
AGENT_CONTEXT_MANIFEST_PATH,
agent_context_staleness,
installed_metaobjects_version,
)
from metaobjects.meta.meta_data import MetaData
from metaobjects.codegen.config import GenConfig
from metaobjects.codegen.overwrite_policy import has_hash_manifest, read_generated_hash
from metaobjects.codegen.project_config import (
CONFIG_FILENAME,
ConfigError,
ProjectConfig,
TargetConfig,
load_project_config,
)
from metaobjects.codegen.generator import Generator
from metaobjects.codegen.generators.entity_model import entity_model
from metaobjects.codegen.generators.extractor_generator import extractor_generator
from metaobjects.codegen.generators.filter_allowlist_generator import (
filter_allowlist_generator,
)
from metaobjects.codegen.generators.names_generator import names_generator
from metaobjects.codegen.generators.output_parser_generator import (
output_parser_generator,
)
from metaobjects.codegen.generators.output_prompt_generator import (
output_prompt_generator,
)
from metaobjects.codegen.generators.router_generator import router_generator
from metaobjects.codegen.generator_registry import (
GENERATOR_REGISTRY,
GeneratorBuildContext,
get_generator,
list_generators,
unsatisfied_requires,
)
from metaobjects.codegen.runner import run_gen
from metaobjects.codegen import eject as owned
from metaobjects.codegen.generators.render_helper_generator import (
_derive_payload_field_tree,
)
from metaobjects.codegen.value_objects import pkg_of, resolve_payload_vo
from metaobjects.meta.template import template_constants as tc
from metaobjects.naming import COLUMN_NAMING_STRATEGIES, DEFAULT_COLUMN_NAMING, package_of_resolution_key
from metaobjects.render.filesystem_provider import FilesystemProvider
from metaobjects.render.verify import (
ERR_REQUIRED_SLOT_UNUSED,
PayloadField,
verify as render_verify,
)
from metaobjects.shared.base_types import TYPE_OBJECT, TYPE_TEMPLATE
#: The error a run with no generator selection reports.
#:
#: There is no default suite. This port used to run EIGHT generators for a caller who
#: named none — entity, router, filter-allowlist, names, payload, output-parser,
#: output-prompt, extractor — which is a shape nobody chose. Java has never had a
#: default set and has been right all along; TypeScript and C# dropped theirs in the
#: same change.
#:
#: Deciding WHICH code an application needs belongs to whoever is building it —
#: increasingly an LLM working in the repo, which is well able to make that call given a
#: truthful catalog and is badly served by a default that pre-empts it. ``--list`` is
#: that catalog.
NO_GENERATORS_SELECTED = (
"gen: no generators selected. Nothing is generated until you choose it — "
"pass --generators <a,b,c>. See the catalog: metaobjects gen --list"
)
class NoGeneratorsSelectedError(ValueError):
"""Raised when a codegen run names no generator. Carries the usage message."""
def __init__(self) -> None:
super().__init__(NO_GENERATORS_SELECTED)
def _resolve_providers(specs: list[str] | None) -> tuple[list[object], list[str]]:
"""Import consumer providers named as ``module:symbol`` (repeatable ``--provider``).
The symbol must be a metaobjects ``Provider`` (or a list of them, or a zero-arg
factory returning either). A provider carries CODE — factories and optional
validators — so it is loaded as a native Python object, not a JSON file (#158).
This is the Python CLI's idiomatic parity with TS ``metaobjects.config.ts``
``providers``; the underlying loader already accepts providers.
"""
if not specs:
return [], []
import importlib
from metaobjects.provider import Provider
providers: list[object] = []
errors: list[str] = []
for spec in specs:
module_name, sep, symbol = spec.partition(":")
if not sep or not module_name or not symbol:
errors.append(
f"--provider '{spec}' must be in 'module:symbol' form "
"(e.g. myapp.providers:my_provider)"
)
continue
try:
module = importlib.import_module(module_name)
except Exception as exc: # ImportError + anything raised at import time
errors.append(f"--provider '{spec}': cannot import '{module_name}': {exc}")
continue
try:
obj = getattr(module, symbol)
except AttributeError:
errors.append(
f"--provider '{spec}': '{module_name}' has no attribute '{symbol}'"
)
continue
# A zero-arg factory is supported: call it to obtain the provider(s).
if callable(obj) and not isinstance(obj, Provider):
try:
obj = obj()
except Exception as exc:
errors.append(f"--provider '{spec}': factory '{symbol}()' raised: {exc}")
continue
for item in obj if isinstance(obj, list) else [obj]:
if isinstance(item, Provider):
providers.append(item)
else:
errors.append(
f"--provider '{spec}': '{symbol}' resolved to "
f"{type(item).__name__}, expected a metaobjects Provider"
)
return providers, errors
def _providers_from_args(args: argparse.Namespace) -> tuple[list[object], bool]:
"""Resolve ``--provider`` specs; print resolution errors. Returns (providers, ok)."""
providers, errors = _resolve_providers(getattr(args, "provider", None))
if errors:
print("error: invalid --provider:", file=sys.stderr)
for msg in errors:
print(f" {msg}", file=sys.stderr)
return providers, False
return providers, True
def _find_config(args: argparse.Namespace) -> Path | None:
"""The config path: ``--config`` if given (even if missing → clear load error),
else ``./metaobjects.config.yaml`` in cwd when it exists, else ``None``."""
explicit = getattr(args, "config", None)
if explicit:
return Path(explicit)
default = Path.cwd() / CONFIG_FILENAME
return default if default.is_file() else None
def _config_providers(config: ProjectConfig) -> tuple[list[object], bool]:
"""Resolve ``config.providers`` with the config file's directory on ``sys.path``.
#267: prepend the config directory so a ``module:symbol`` provider living beside
the config imports with no ``PYTHONPATH=``. Idempotent; the entry is left in
place (a short-lived CLI process). Prints resolution errors; returns (providers, ok).
"""
config_dir = str(config.config_dir)
if config_dir not in sys.path:
sys.path.insert(0, config_dir)
providers, errors = _resolve_providers(config.providers)
if errors:
print("error: invalid provider in config:", file=sys.stderr)
for msg in errors:
print(f" {msg}", file=sys.stderr)
return providers, False
return providers, True
def _select_targets(
config: ProjectConfig, target_name: str | None
) -> tuple[list[TargetConfig], str | None]:
"""All targets, or the single ``--target`` (error string when the name is unknown)."""
if target_name is None:
return config.targets, None
t = config.target(target_name)
if t is None:
known = ", ".join(sorted(x.name for x in config.targets))
return [], f"unknown --target {target_name!r}; known targets: {known}"
return [t], None
def _merge_libraries(*groups: "list[str] | tuple[str, ...] | None") -> list[str]:
"""Union several ``libraries`` selection-token groups, de-duplicated and
order-preserving (first occurrence wins the position).
Every group is a selection from a DIFFERENT source for the same concept —
the neutral ``.metaobjects/config.json`` (FR-043 §12 Q4, read at every rung)
and, where one exists, a per-port config surface's own ``libraries:`` key
(Python's ``metaobjects.config.yaml``). Neither is authoritative over the
other, so a project may declare libraries in either place or both.
"""
seen: dict[str, None] = {}
for group in groups:
for token in group or ():
seen.setdefault(token, None)
return list(seen)
def _load_root(
metadata_dir: str,
strict: bool = False,
providers: list[object] | None = None,
libraries: list[str] | None = None,
unknown_attr_out: list[str] | None = None,
) -> tuple[MetaData | None, list[str]]:
"""Load metadata; return ``(root, error_messages)``. ``root`` is None on error.
``unknown_attr_out`` — when given on a LENIENT load, a second strict load of the
same metadata is run and its ``ERR_UNKNOWN_ATTR`` findings are appended, formatted,
so ``gen`` can warn about what ``verify`` would reject instead of staying silent.
``strict`` (ADR-0023, #96) — when True, an undeclared own ``@attr`` →
``ERR_UNKNOWN_ATTR``. Defaults False so ``gen`` / ``docs`` keep the legacy
open-attr load; only ``verify`` opts in (strict-by-default with a ``--lax``
escape).
``providers`` (#158) — consumer providers (from ``--provider module:symbol``)
composed ON TOP of the core set, so a config-registered custom subtype resolves
through the standalone CLI just as it does when an app loads the loader directly.
Empty/None keeps the core-providers-only default (``from_directory`` convenience).
``libraries`` — MetaObjects-shipped library packages (config key ``libraries``,
e.g. ``["ai"]``). Without this the CLI could not load ``metaobjects::ai::LlmCallBase``,
so ``metaobjects gen`` failed with ERR_UNRESOLVED_SUPER on exactly the metadata the
registered ``trace-helper`` generator exists to consume — the generator was reachable
from the CLI while its input was not. ADDITIVE to (never a replacement for) the
neutral ``.metaobjects/config.json``'s own ``libraries`` key (FR-043 §12 Q4), which
this function ALWAYS also reads, from the project holding ``metadata_dir``
(:func:`project_root_for`) — an explicit-``metadata_dir`` CLI invocation
(``gen``, ``docs``, ``verify --templates``, ``verify --codegen``'s
regenerate-to-temp path) never resolves a `Collection`
(:func:`resolve_metadata_location`'s docstring: rung 1 is not its concern), so
without this an adopter whose opt-in lives only in ``.metaobjects/config.json``
could not run any explicit-directory command against metadata using a library
at all — exactly the gap ``config``-mode (:func:`_cmd_gen_config`) does not have,
because it already threads ``config.libraries`` through
:func:`_load_root_from_collection`.
"""
try:
neutral = read_neutral_config(project_root_for(metadata_dir))
except ParseError as exc:
return None, [f"{exc.code}: {exc}"]
all_libraries = _merge_libraries(neutral.libraries if neutral is not None else None, libraries)
def load(strict_load: bool) -> LoadResult:
if providers:
from metaobjects.core_types import core_providers
return MetaDataLoader.from_directory(
metadata_dir,
providers=[*core_providers, *providers],
strict=strict_load,
libraries=all_libraries,
)
return MetaDataLoader.from_directory(
metadata_dir, strict=strict_load, libraries=all_libraries
)
try:
result = load(strict)
if unknown_attr_out is not None and not strict and not result.errors:
unknown_attr_out.extend(_unknown_attr_findings(load(True)))
except OSError as e:
# `DirectorySource.expand()` walks via `iterdir()` (not `rglob()`, so it
# can follow a symlinked subdirectory — the I1 fix) and raises a plain
# OSError (FileNotFoundError / NotADirectoryError, or `SymlinkLoopError`
# on a symlink cycle) when `metadata_dir` itself can't be walked. Nothing
# upstream of this call catches it, so it would otherwise surface as a
# raw Python traceback instead of the coded "error: failed to load
# metadata" every other `_load_root` failure prints. Mirrors the
# TypeScript reference, which wraps the equivalent `readdir` failure in
# a clean `Error` at the same boundary (`metadata-files.ts`,
# `listMetadataFiles`: "cannot read metadata directory ...").
return None, [f"cannot read metadata directory {metadata_dir}: {e}"]
if result.errors:
msgs = [f"{e.code}: {e.message}" for e in result.errors]
return None, msgs
try:
# FR-043 §3.4 / §3.5 — same misuse guard the `Collection` load path runs
# (`_load_collection_result`): an explicit-directory load opted into a
# library was the one path that never checked for an ejected-copy
# collision or an adopter node declared straight into the library's
# package.
refuse_library_package_misuse(result.root, all_libraries)
except ParseError as exc:
return None, [f"{exc.code}: {exc}"]
return result.root, []
def _load_collection_result(
collection: Collection,
strict: bool = False,
providers: list[object] | None = None,
libraries: list[str] | None = None,
) -> LoadResult:
"""Load a resolved `Collection`'s files — its dependency artifacts leading,
each under its `dep:<name>/<artifact>` source id (FR-023) — then run the
post-load ownership refusal (`ERR_DEPENDENCY_PACKAGE_NOT_OWNED`, DESIGN
§11.5). Mirrors the TS `loadMemory` (`sdk/src/memory.ts`).
``libraries`` is ADDITIVE to ``collection.libraries`` (the neutral
`.metaobjects/config.json`'s own key, read at every rung — FR-043 §12 Q4),
never a replacement for it: a caller passes a per-port config surface's own
selection here (Python's `metaobjects.config.yaml` `libraries:` key), and
both are merged before resolving library sources.
Lower-level than :func:`_load_root_from_collection`: this returns the raw
`LoadResult` (structured `MetaError`s, a code + provenance on each) rather
than the CLI's flattened `(root, message-strings)` contract, so a caller
that needs the CODE or the offending FILE — the dependency corpus runner —
can read them. ``refuse_unowned_packages`` runs AFTER the loader's own
errors, never before: an unflagged overlay whose target the upstream
removed fails first, with its own coded error.
"""
# `collection.libraries` (FR-043 §12 Q4 — the neutral `.metaobjects/config.json`,
# read at every rung) is merged with whatever the caller additionally passes
# (a per-port config surface's own `libraries:` key, e.g. Python's
# `metaobjects.config.yaml` — see `_cmd_gen_config`). Neither source is
# authoritative over the other.
all_libraries = _merge_libraries(collection.libraries, libraries)
lib_sources: list[object] = []
if all_libraries:
from metaobjects.library import library_sources
lib_sources = library_sources(all_libraries)
sources = [FileSource(p, id=collection.file_ids.get(p)) for p in collection.files]
if providers:
from metaobjects.core_types import core_providers
loader = MetaDataLoader(providers=[*core_providers, *providers], strict=strict)
else:
loader = MetaDataLoader(strict=strict)
result = loader.load([*lib_sources, *sources])
if not result.errors:
refuse_unowned_packages(result.root, collection.imported_packages, collection.imported_nodes)
# FR-043 §3.4 / §3.5 — the same rule for a shipped LIBRARY's package, where the
# two ways to get it wrong are opposite: a node the library also declares (an
# ejected copy, still opted in) and one it does not (a new node in someone
# else's package).
refuse_library_package_misuse(result.root, all_libraries)
return result
def _load_root_from_collection(
collection: Collection,
strict: bool = False,
providers: list[object] | None = None,
libraries: list[str] | None = None,
unknown_attr_out: list[str] | None = None,
) -> tuple[MetaData | None, list[str]]:
"""Load a resolved `Collection` — the source-resolution ladder's
``.metaobjects/config.json`` rung (:func:`resolve_metadata_location`), which
can resolve to several directories, individual files, AND a dependency's
snapshot artifact, none of which a single ``from_directory`` call can
express. Mirrors :func:`_load_root`'s ``strict``/``providers``/``libraries``
contract exactly (CLI-facing: flattened error messages, never a raised
exception for a plain loader error).
"""
try:
result = _load_collection_result(collection, strict=strict, providers=providers, libraries=libraries)
except ParseError as exc:
return None, [f"{exc.code}: {exc}"]
if result.errors:
msgs = [f"{e.code}: {e.message}" for e in result.errors]
return None, msgs
if unknown_attr_out is not None and not strict:
# See `_load_root`: what a strict load (`verify`) would reject, for `gen` to warn.
try:
strict_result = _load_collection_result(
collection, strict=True, providers=providers, libraries=libraries
)
except ParseError:
strict_result = None
if strict_result is not None:
unknown_attr_out.extend(_unknown_attr_findings(strict_result))
return result.root, []
#: Trailing advice after ``gen``'s unknown-attribute warnings. ``gen`` has no strict
#: flag of its own; ``verify`` is the strict door (ADR-0023), so that is what it names.
UNKNOWN_ATTR_GEN_ADVICE = (
"metaobjects gen loads leniently and generated anyway, but `metaobjects verify` "
"rejects this metadata (ADR-0023): fix or remove the attribute - a typo'd one "
"(`isAbstrakt`, `requird`) silently changes what is generated."
)
def _format_unknown_attr(err: MetaError) -> str:
"""One ``gen`` warning: code + message (naming attribute and node), then the file."""
env = getattr(err, "envelope", None)
files = [f for f in (getattr(env, "files", None) or ()) if f]
json_path = getattr(env, "json_path", None)
where = " ".join(
p for p in (", ".join(files) if files else None, f"at {json_path}" if json_path else None) if p
)
head = f"{err.code.value if hasattr(err.code, 'value') else err.code}: {err.message}"
return f"{head}\n in {where}" if where else head
def _unknown_attr_findings(result: LoadResult) -> list[str]:
"""The ERR_UNKNOWN_ATTR findings of a STRICT load, formatted for ``gen``."""
return [
_format_unknown_attr(e) for e in result.errors
if (e.code.value if hasattr(e.code, "value") else e.code) == ErrorCode.ERR_UNKNOWN_ATTR.value
]
def _print_unknown_attr_warnings(findings: list[str]) -> None:
"""``gen`` loads leniently; an unknown attribute ``verify`` would reject is named here
as a warning (attribute, node, file). Advisory only: the exit code is unchanged."""
if not findings:
return
for f in findings:
print(f"warning: {f}", file=sys.stderr)
print(f"warning: {UNKNOWN_ATTR_GEN_ADVICE}", file=sys.stderr)
def _strict_load_hint() -> str:
"""Actionable next-steps when strict verify rejects an undeclared @attr."""
return (
"verify is strict (ADR-0023): every authored @attr must be declared. Fix: "
"register the attr on a metadata provider, OR move arbitrary author-supplied "
"properties into an `attr.properties` bag, OR re-run with `--lax` to keep the "
"legacy open-attr load."
)
#: The canonical directory name for authored template bodies. Named `prompts` because
#: that is what the Node CLI has always called it (`DEFAULT_PROMPTS_DIR`), and the Node
#: CLI is the schema door every project meets first.
DEFAULT_PROMPTS_DIR = "prompts"
#: The name this port defaulted to before 1.0.5, kept as the FALLBACK rather than
#: replaced. Flipping the default outright would move where an existing project's
#: template refs resolve from, and a project whose bodies are in `templates/` would
#: start finding nothing — a silent drift-gate failure, or worse a `verify --codegen`
#: that convicts committed output. A fallback breaks nobody: a project with
#: `templates/` and no `prompts/` behaves exactly as it did.
LEGACY_TEMPLATES_DIR = "templates"
def _default_template_root(base: str | Path = ".") -> str:
"""The template root to use when the caller named none.
`prompts` when it exists, else `templates`. The two halves of the toolchain
disagreed about this: the Node CLI has always defaulted to `prompts`, while this
port and C#'s `GenCommand` defaulted to `templates`, so a project following the
Node CLI's layout — which this repo's own adopter estate does — had a `gen` that
looked somewhere its `prompts/` was not.
"""
return (DEFAULT_PROMPTS_DIR if Path(base, DEFAULT_PROMPTS_DIR).is_dir()
else LEGACY_TEMPLATES_DIR)
def _template_root_for(args: argparse.Namespace) -> str:
"""The on-disk template root for THIS invocation. Never ``None``.
EVERY path that builds a generator list must call this, for the same reason the C#
port says it of its template-spec resolution: ``gen`` and ``verify --codegen``
regenerate the same suite and diff the result, so if they resolve the template root
differently, ``verify`` builds ``render-helper`` against another directory and
reports the committed helpers as stale — with a remedy that loops, since
regenerating cannot fix a disagreement about where the templates are. That is also
why the fallback applies to BOTH commands: a default on one and not the other is the
same disagreement wearing a different hat.
The two commands spell the directory differently, and that is not cosmetic. On
``gen`` it is ``--templates <dir>``. On ``verify`` ``--templates`` is the BOOLEAN
prompt-drift subverb and the directory is ``--prompts <dir>`` (F101's converged
spelling; ``--templates-root`` is its deprecated alias, and argparse lands both on
``templates_root``). Hence the isinstance check rather than a plain ``or`` chain:
``getattr(args, "templates")`` is a ``True`` on one of these commands, and ``or``
would hand a boolean to a path join.
"""
gen_dir = getattr(args, "templates", None)
if isinstance(gen_dir, str) and gen_dir:
return gen_dir
explicit_verify = getattr(args, "templates_root", None)
if explicit_verify:
return explicit_verify
return _default_template_root()
def _resolve_generators(
names: str, ctx: GeneratorBuildContext | None = None, owned_root: Path | None = None,
) -> tuple[list[Generator], list[str]]:
"""Resolve a comma-separated list of STABLE generator names via the registry.
An entry of the form ``module:symbol`` is an adopter-owned copy (``metaobjects
eject``), imported with *owned_root* (default: the working directory) on ``sys.path``
— the config directory in config mode, so a copy beside the config resolves.
Returns ``(generators, errors)``. An unknown name produces a clear error and
no generators (so the caller can fail with exit code != 0).
*ctx* carries what a factory needs beyond the metadata — today the on-disk template
root, which ``render-helper`` requires for its build-time drift gate. It used to be
absent entirely: the registry hardcoded ``template_root="templates"``, so
``--templates`` reached the template-spec pass and nothing else, and selecting
``render-helper`` read a directory the user had never named.
"""
if ctx is None:
ctx = GeneratorBuildContext()
requested = [n.strip() for n in names.split(",") if n.strip()]
gens: list[Generator] = []
errors: list[str] = []
selected: list[str] = []
for n in requested:
entry = get_generator(n)
if entry is None and (":" in n or "." in n):
gen, err = owned.build_owned(n, owned_root or Path.cwd(), ctx)
if err is not None:
errors.append(err)
continue
gens.append(gen)
selected.append(gen.name)
continue
if entry is None:
known = ", ".join(sorted(GENERATOR_REGISTRY))
errors.append(f"unknown generator {n!r}; known: {known}")
continue
gens.append(entry.factory(ctx))
selected.append(n)
if not errors and not gens:
errors.append("no generators selected (empty --generators list)")
if not errors:
for warning in unsatisfied_requires(selected):
print(f"warning: {warning}", file=sys.stderr)
return gens, errors
def _parse_entities(value: str | None) -> list[str] | None:
"""Parse a comma-separated ``--entities`` value into a name list (``None`` =
no filter = generate every entity). Blank names are dropped."""
if not value:
return None
names = [n.strip() for n in value.split(",") if n.strip()]
return names or None
def _refuse_imported_entities(
entity_names: list[str] | None,
root: MetaData,
collection: Collection,
) -> str | None:
"""FR-023 §11.1 item 2 — the error message for a target's ``entities:`` (or
``--entities``) naming ONLY objects imported from a dependency and excluded
from output (imported metadata is load-only by default; only the project's
own `scope.include` naming the package literally opts it in). Returns
`None` for every other case, INCLUDING a name matching nothing at all —
that stays the runner's own "no entities matched" warning, unchanged.
Mirrors the TS `refuseImportedPositional` (`cli/src/commands/gen.ts`).
A named entity is a bare NAME, not a fully-qualified one, so two packages
declaring the same short name are both candidates; the refusal fires only
when NONE of them would generate.
"""
if not entity_names:
return None
# ADR-0039 sanctioned own: top-level object scan on the loader ROOT
# (metadata.root is never extended, so own == effective) — mirrors
# `codegen.runner._objects`.
all_objects = [
c for c in root.own_children() if c.type == TYPE_OBJECT and isinstance(c, MetaObject)
]
for name in entity_names:
matches = [o for o in all_objects if o.name == name]
if not matches:
continue # nothing loaded by this name — the runner's own warning covers it
all_excluded = all(
collection.imported(o.resolution_key()) and not collection.in_scope(o.resolution_key())
for o in matches
)
if not all_excluded:
continue
fqn = matches[0].resolution_key()
pkg = package_of_resolution_key(fqn)
dep_name = imported_from(pkg, collection.dependencies) or pkg
return (
f"error: '{name}' ({fqn}) is imported from dependency '{dep_name}' and is not "
"generated here — add its package to scope.include in .metaobjects/config.json"
)
return None
def _run_suite(
root: MetaData,
out_dir: str,
generators: list[Generator] | None = None,
entity_filter: list[str] | None = None,
emit_package_init: bool = True,
gen_state_dir: str | None = None,
column_naming: str = DEFAULT_COLUMN_NAMING,
project_root: str | None = None,
baseline: str = "default",
refused_out: list[str] | None = None,
select: Callable[[str], bool] | None = None,
) -> list[str]:
"""Run a generator suite against an ALREADY-LOADED ``root`` into ``out_dir``.
Split out from :func:`_generate` so a single gen invocation can load the
metadata once and run multiple passes against it (the default Python suite +
the ``--template-spec`` pass). See :func:`_generate` for the parameter
semantics. Returns the written paths.
``select`` (FR-023 §11.1 item 2) — normally a resolved `Collection`'s
`in_scope`: the output filter that excludes an imported dependency's
objects from codegen unless the project's own `scope.include` names their
package. `None` (the default) admits everything, byte-identical to a
project with no dependencies.
NOTE: ``run_gen``'s output-path collision guard is per-pass — a
``--template-spec`` ``outputPattern`` that collides with a default-suite path
is not flagged across the two passes (it would overwrite, since default files
carry the ``@generated`` header). Accepted marginal limitation (requires user
misconfiguration).
"""
# The hash manifest gives `metaobjects gen` hand-edit detection: without it the
# write path falls back to the legacy @generated-marker rule, which cannot tell an
# edited generated file from a pristine one (an edited file keeps its marker).
# COMMIT `.gen-state/.hashes.json`.
#
# Anchored on the caller's project, NEVER on `Path.cwd()`. cwd is whatever directory
# the process happens to sit in, so a cwd default leaks a `.metaobjects/` into any
# package that merely runs a test through this function — the exact leak codegen-ts's
# runner refuses, for the same reason. No project ⇒ no manifest ⇒ documented weaker
# guarantees, rather than state scattered somewhere arbitrary.
config = GenConfig(
out_dir=out_dir,
emit_package_init=emit_package_init,
gen_state_dir=gen_state_dir,
column_naming=column_naming,
# Manifest keys are relative to the PROJECT, which is what the manifest is
# anchored on — not to ``--out``, which varies per run while the manifest does
# not. Passed alongside gen_state_dir because the two are one decision: state
# that lives in the project must be keyed by the project.
project_root=project_root,
# `--baseline=adopt`: record what is on disk and write nothing. The refusal it
# exists for is the pre-manifest one, so it reaches every real gen path and no
# verify path (those pass gen_state_dir=None and record nothing at all).
baseline=baseline,
)
# No default suite — see NO_GENERATORS_SELECTED. A caller that names none gets a
# usage error and an empty out dir, never a shape this CLI picked.
if not generators:
raise NoGeneratorsSelectedError
suite = generators
result = run_gen(config, root, generators=suite, entity_filter=entity_filter, select=select)
for warning in result.warnings:
print(f"warning: {warning}")
# A refusal is a FAILED gate, and the return value carries only what was WRITTEN — so
# without this the one fact a CI step reads (the exit code) could not see it. An
# out-list rather than a changed return type: five call sites read the written paths,
# and only the ones that own an exit code care about the refusals.
if refused_out is not None:
refused_out.extend(path for path, status in result.files if status == "refused")
return [path for path, status in result.files if status != "refused"]
def _generate(
metadata_dir: str,
out_dir: str,
generators: list[Generator] | None = None,
entity_filter: list[str] | None = None,
emit_package_init: bool = True,
strict: bool = False,
providers: list[object] | None = None,
column_naming: str = DEFAULT_COLUMN_NAMING,
) -> tuple[list[str], list[str]]:
"""Run the generator suite into ``out_dir``.
``generators`` defaults to the zero-config default suite; pass a registry-
resolved subset for ``--generators``. ``entity_filter`` (the ``--entities``
allowlist) restricts which entities are EMITTED while the WHOLE model is still
loaded, so cross-entity references (``extends`` bases, ``@objectRef`` VOs)
resolve — emit a subset without splitting the metadata. ``emit_package_init``
is ``False`` for the format-agnostic ``--template-spec`` generators (whose
output is text/markdown/csv/json/xml/html, never a Python package) so no
spurious ``__init__.py`` is scattered through their output tree. Returns
``(written_paths, errors)``. On a load error, ``errors`` is non-empty and no
files are written. ``strict`` (ADR-0023, #96) is threaded to the load — the
``verify --codegen`` caller passes True; ``gen`` keeps the default False.
``column_naming`` (R31) must match the strategy the committed output was
produced with, or the ``names`` generator's constants regenerate under the
wrong strategy and every ``<ENTITY>_<FIELD>_COLUMN`` reports spurious drift —
the caller reads it from the SAME ``--column-naming`` flag `gen` accepts.
"""
root, errors = _load_root(metadata_dir, strict=strict, providers=providers)
if root is None:
return [], errors
# The project root is the metadata dir's parent — the directory holding
# `metaobjects/`, which is where `.metaobjects/` belongs beside it.
# verify --codegen is the ONLY caller of _generate, and it regenerates into a
# throwaway temp dir purely to diff. It must NOT record a manifest: that would
# mutate the user's project from a read-only drift check, with keys relative to a
# directory deleted seconds later. The real gen paths anchor their own manifest via
# gen_state_dir_for().
return (
_run_suite(
root, out_dir, generators, entity_filter, emit_package_init, None,
column_naming=column_naming,
),
[],
)
def gen_state_dir_for(metadata_dir: str) -> str:
"""Where a project's codegen hash manifest lives.
Anchored on the project — never on ``Path.cwd()``, which is whatever directory the
process happens to sit in and would scatter ``.metaobjects/`` into any package that
merely runs a generator.
It resolves the project through :func:`project_root_for` rather than repeating
``Path(metadata_dir).resolve().parent``. That function's own docstring already claimed
this one "is anchored here" while the code spelled the rule out a second time, and the
manifest key and the manifest LOCATION have to agree on where the project is or a
lookup finds nothing and every file reads as "not ours".
"""
return str(project_root_for(metadata_dir) / ".metaobjects" / ".gen-state")
#: The conventional declarative-template-spec file (SP-1 §4), discovered when
#: ``--template-spec`` is not passed. VISIBLE and at the project root on purpose:
#: this repo splits author input (visible — ``metaobjects/``, ``templates/``) from
#: tool state (hidden — ``.metaobjects/``), and a spec is authored, not generated.
#: It also sits beside the ``templates/`` dir its refs resolve under.
TEMPLATE_SPEC_FILENAME = "template-spec.json"
def project_root_for(metadata_dir: str) -> Path:
"""The project a metadata directory belongs to: its PARENT — the directory
holding ``metaobjects/``. Same rule as :func:`gen_state_dir_for` (which is
anchored here) and the C# port's ``GenCommand.ProjectRootFor``."""
return Path(metadata_dir).resolve().parent
def template_spec_path_for(metadata_dir: str, explicit: str | None) -> str | None:
"""Resolve the template-spec to use, or ``None`` for "no template generators".
An explicit ``--template-spec`` always wins and is used verbatim (a flag naming
a missing file stays a hard error at read time — silently ignoring a path the
user typed would be worse). Otherwise the conventional
``<projectRoot>/template-spec.json`` is used IF it exists.
**Every code path that builds a generator list must call this.** The defect this
exists to prevent is `gen` and `verify --codegen` resolving the spec differently:
`gen` honoured the flag while `verify` built its own list and never looked, so
`verify` regenerated WITHOUT the template generators and reported their committed
output as stale — with a remedy that loops (regenerating cannot produce files the
regen does not know about).
"""
if explicit:
return explicit
candidate = project_root_for(metadata_dir) / TEMPLATE_SPEC_FILENAME
return str(candidate) if candidate.is_file() else None
def template_spec_generators(
metadata_dir: str, explicit: str | None, templates_dir: str,
) -> tuple[list[Generator], str | None]:
"""The declarative Mustache generators for this project, or an error message.
Returns ``(generators, None)`` on success — an EMPTY list when there is no spec
at all — and ``([], message)`` when a spec was found but could not be used.
"""
spec_path = template_spec_path_for(metadata_dir, explicit)
if spec_path is None:
return [], None
from metaobjects.codegen.template_codegen.template_spec import (
parse_template_spec,
template_spec_to_generators,
)
try:
spec = parse_template_spec(json.loads(Path(spec_path).read_text(encoding="utf-8")))
except (OSError, ValueError) as exc:
return [], f"error: invalid template-spec {spec_path!r}: {exc}"
return template_spec_to_generators(spec, FilesystemProvider(templates_dir)), None
def resolve_metadata_location(
config: ProjectConfig | None,
root: Path,
) -> Collection:
"""The precedence ladder for where metadata lives, rungs 2-4. First match wins.
Returns the full FR-023-aware `Collection` at EVERY rung — `dependencies` (and
`scope`/`migrate.scope`) are read from `root`'s neutral `.metaobjects/config.json`
regardless of which rung supplied the OWN files (DESIGN §2.3).
2. This port's native surface — ``metadata`` in ``metaobjects.config.yaml``.
3. ``sources`` in the port-neutral ``.metaobjects/config.json``.
4. The built-in default directory.
Rung 1 — an explicit CLI argument (the positional ``metadata_dir``) — is
NOT this function's concern: every command handler that accepts one already
loads it directly via ``_load_root(args.metadata_dir)`` (``MetaDataLoader
.from_directory``, a single-directory load with no resolve-then-rejoin step
of its own) BEFORE this function would even be reachable, so this function is
only ever called with rung 1 already having been tried and found absent. An
earlier revision accepted an ``explicit`` parameter and duplicated rung 1
here via `resolve_sources` — reachable from no command handler, so it could
silently drift from the real rung-1 path (e.g. picking up
`source_resolver.py`'s CLI-facing `_pending` exclusion while
`_load_root`'s loader-level walk does not) with nothing to notice. Removed
rather than wired: replumbing every metadataDir-taking command onto
`resolve_sources`'s `from_uris`-based load instead of `from_directory` is a
materially different, higher-risk change than this ladder needs.
A file that EXISTS at any rung but is malformed raises rather than falling
through to the next rung. See
`docs/superpowers/specs/2026-08-19-cross-port-metadata-sources-design.md` §5.
"""
from metaobjects.config.source_resolver import (
build_collection,
resolve_collection_full,
resolve_sources,
)
if config is not None:
# `config.metadata_dir()` is already resolved to an absolute path
# (`ProjectConfig._resolve_under`), so the base passed here is
# likewise irrelevant to resolving OWN files — but `root` is still
# where `.metaobjects/config.json` (dependencies/scope) is read from.
own_files = resolve_sources(root, [{"path": config.metadata_dir()}])
return build_collection(root, own_files)
# Rungs 3 and 4 both live in `resolve_collection_full`.
return resolve_collection_full(root)
def _resolve_metadata_location_or_print_error(
config: ProjectConfig | None, root: Path
) -> Collection | None:
"""``resolve_metadata_location`` for the no-explicit-``metadata_dir`` CLI
paths (``docs``, and the ``gen``/``verify --codegen`` neutral fallbacks),
translating a raised ``ParseError`` into this CLI's print-and-return-1
convention instead of letting it propagate. Returns ``None`` on failure —
the caller has nothing further to print and should return 1.
"""
try:
return resolve_metadata_location(config=config, root=root)
except ParseError as exc:
print(f"error: could not resolve metadata location: {exc}", file=sys.stderr)
return None
#: The default api-surface subdir (the cross-port contract's ``api/python``).
_DOCS_DEFAULT_API_SUBDIR = "api/python"
def _cmd_docs(args: argparse.Namespace) -> int:
"""``docs`` — emit the generated Python SDK api surface (the ``api/python``
surface of the cross-port SDK-docs contract) under ``--out``.
Wiring only: every symbol name comes from the ``PythonApiModelBuilder`` (which
keys off the naming seam + each generator's applies-predicate) and every path
comes from :mod:`metaobjects.apidocs.paths` — nothing is re-derived here. Output
paths are collision-checked (duplicate page path → failure) BEFORE anything is
written, mirroring the C# ``DocsCommand`` / the Java Mojo. Layout is ``package``
(the cross-port contract). ``--model-base-url`` federates the model back-links
when set."""
from metaobjects.apidocs import (
Layout,
PythonApiDocsRenderer,
PythonApiModelBuilder,
doc_page_output_path,
model_cross_href,
)
providers, providers_ok = _providers_from_args(args)
if not providers_ok:
return 1
if args.metadata_dir is None:
# Mirrors `gen`'s no-positional routing (#267): rung 2 (`metadata` in
# `metaobjects.config.yaml`, if one is found), else rungs 3-4 via
# `resolve_metadata_location` (the neutral `.metaobjects/config.json`
# `sources` key this feature adds, else the built-in default
# directory). Before this, `metadata_dir` was a REQUIRED positional
# here, so this ladder was unreachable from `docs` even though `gen`
# and `verify --codegen` could both already reach it.
root_dir = Path.cwd()
config: ProjectConfig | None = None
config_path = _find_config(args)
if config_path is not None:
try:
config = load_project_config(config_path)
except ConfigError as exc:
print(f"error: {exc}", file=sys.stderr)
return 1
collection = _resolve_metadata_location_or_print_error(config, root_dir)
if collection is None:
return 1
docs_libraries: list[str] | None = None
if config is not None:
# `config` was already loaded above (to resolve rung 2's `metadata`
# key) — its OWN `providers`/`libraries` must reach this load too,
# exactly as `gen`'s config-mode load does (`_cmd_gen_config`), or a
# project relying on either (e.g. `libraries: ["ai"]` to resolve
# `extends: metaobjects::ai::LlmCallBase`) fails `docs` with
# `ERR_UNRESOLVED_SUPER` while `gen` on the same config succeeds.
config_providers, config_providers_ok = _config_providers(config)
if not config_providers_ok:
return 1
providers = [*providers, *config_providers]
docs_libraries = config.libraries
root, errors = _load_root_from_collection(
collection, providers=providers, libraries=docs_libraries
)
project_default = root_dir.name
else:
root, errors = _load_root(args.metadata_dir, providers=providers)
project_default = Path(args.metadata_dir).resolve().name
if root is None:
print("error: failed to load metadata:", file=sys.stderr)
for msg in errors:
print(f" {msg}", file=sys.stderr)
return 1