feat: new manifest format

[atilafassina]

Summary

Evolves the AppKit plugin manifest contract from v1.0 to v2.0 and swaps the canonical authoring surface from hand-written JSON Schema to Zod via the Standard Schema interface. AJV, the json-schema-to-typescript codegen pipeline, and the standalone semantic-validator delete; refinement and .transform() on Zod schemas replace them.

Mid-review (2026-05-18) the design picked up a substitutability-gate amendment that deleted postScaffold entirely in favor of a unified plugin-level scaffolding.rules block, and audited every prose rule against the gate. See "2026-05-18 amendment" below.

Important

~91% of insertions are auto-regenerated artifacts — two JSON Schema files (docs/static/schemas/{plugin-manifest,template-plugins}.schema.json) account for +10,400 lines.

.gitattributes marks all generated artifacts linguist-generated=true. GitHub collapses them in the file list — substantive review surface is ~32 files / ~2,300 insertions.

What changes vs main

Contract

• discoveryDescriptor is a discriminated union: { type: "kind", resourceKind, ... } | { type: "cli", cliCommand, ... }. Six known kinds: warehouse, genie_space, volume, postgres_project, postgres_branch, postgres_database.
• cli variant rejects shell metacharacters (;, |, &, backtick, $, newlines) on cliCommand and shortcut. .describe() notes direct authors to kind for first-party resources and flag the cli shape as minimal-and-may-tighten.
• RESOURCE_KIND_COMMANDS map next to the schema; commands verified against Databricks CLI v0.299.0. volume kind carries parents: ["catalog", "schema"] — declares free-text prompts the runner must collect before invoking discovery (replaces the prior prose rule).
• origin on template fields is a .transform() output (drift-impossible).
• rules.must / rules.should / rules.never items capped at maxLength: 120. Per-bucket and cross-bucket dedup enforced via .superRefine().
• 4 core plugin manifests (analytics, files, genie, lakebase) migrated to the kind variant; lakebase gains a project field.

Authoring surface

• New: packages/shared/src/schemas/manifest.ts (Zod schemas + RESOURCE_KIND_COMMANDS + TEMPLATE_SCAFFOLDING).
• New: tools/generate-json-schema.ts (Zod → draft-07 JSON Schema, emits to docs/static/schemas/).
• New runtime dep on packages/shared: @standard-schema/spec.

Deletions

• Deps removed: ajv, ajv-formats, json-schema-to-typescript.
• packages/shared/src/schemas/{plugin-manifest.schema.json,template-plugins.schema.json,plugin-manifest.generated.ts}.
• tools/generate-schema-types.ts, docs/scripts/copy-schemas.ts.
• validate-manifest.ts (498 → 177 lines): AJV plumbing + runSemanticValidation + helpers.
• enrichFieldsWithOrigin, validateDiscoveryOrigin.
• postScaffoldStepSchema and postScaffold field (post-amendment — see below).

Migrated to Zod-direct

• schema-resources.ts and tools/generate-registry-types.ts (no more runtime JSON-schema reads).
• manifest-types.ts is now a re-export shim (z.infer types + StandardSchemaV1).

Plugin Rules (per core plugin, scaffolding.rules)

Plugin	must	should
analytics	Before init, ensure the SQL Warehouse passed via --set analytics.sql-warehouse.id is running	After init, ensure config/queries/ has at least one .sql file before running npm run typegen
files	Before init, verify your Unity Catalog volume exists and you have WRITE_VOLUME permission	—
genie	After init, configure the 'spaces' map in plugin config with alias-to-Space-ID mappings	—
jobs	(no rules)	(no rules)
lakebase	After init, run database migrations via 'pnpm drizzle:migrate' before first request	After init, verify Lakebase connectivity with 'psql $PGHOST -c "select 1"'

Template-level scaffolding.rules

{
  "must": [
    "Keep all secrets and credentials only in app.yaml, databricks.yml, and/or .env"
  ],
  "should": [
    "ask user when in doubt of resource to use for plugin"
  ],
  "never": [
    "guess resources when multiple or no options are available",
    "embed secrets in files that will go to the client-bundle"
  ]
}

Each entry passes the substitutability gate (describes a cross-cutting agent behavior at decision time, not derivable from structured manifest data). Each ≤120 chars. Conflict-detection-safe against the current plugin rule corpus (no must ↔ never overlaps).

2026-05-18 amendment — substitutability gate

The original v2.0 plan shipped postScaffold arrays ({ instruction, required }) at the plugin level. During v2.0 implementation we found the format couldn't participate in the databricks-apps agent skill's command-construction flow — prose doesn't compose across plugins, duplicates work the skill drives from structured resources, and { instruction, required } adds zero parseable signal.

The amendment introduces one design principle and applies it uniformly:

A prose rule in the manifest is legitimate only if it cannot be expressed as structured data on a resource, plugin config, or template.

Changes that landed under the amendment:

• postScaffold deleted. postScaffoldStepSchema and the postScaffold field removed from both pluginManifestSchema and the template plugin schema. All four core plugins migrated to plugin-level scaffolding.rules.
• Plugin-level scaffolding.rules with must / should / never arrays (≤120 chars each, no duplicates within a bucket, no overlap across buckets — all enforced via .superRefine()).
• Template-level scaffoldingRulesSchema gains should[]? for parity with plugin-level.
• TEMPLATE_SCAFFOLDING audit — original 4 must entries pruned (every one was substitutable: skill content, derivable from requiredByTemplate, derivable from field.env, or a discovery-descriptor concern). Original 3 never entries replaced with 2 sharper rules covering credentials hygiene and resource-selection decision points; one entry deleted as unreachable. See "Template-level scaffolding.rules" above.
• A6 schema-extension decisions:
  • (a) Volume catalog+schema prerequisite → EXTEND: RESOURCE_KIND_COMMANDS.volume.parents = ["catalog", "schema"]. Models transient query inputs as a kind-level property without conflating with dependsOn (which references sibling fields).
  • (b) Config-driven --set derivation for genie spaces → ACCEPT AS PROSE pending Track B confirmation of the skill's consumption model.
  • (c) Liveness preconditions (analytics warehouse) → ACCEPT AS PROSE. Runtime state checks aren't structural data.
• Per-plugin rule audit: 7 substitutability-gate violations introduced during experimentation were caught and reverted (permission duplications, resource-existence tautologies, inactionable-at-scaffold-time directives). Net effect: zero new plugin rules survive the gate that weren't already in the post-Phase-1 baseline.

.claude/commands/ skill extensions

Outside the packages/ tree but relevant to reviewers — the in-repo plugin commands gained procedural v2.0 semantic checks:

• audit-core-plugin.md — Step 3.6 (3.6a substitutability-gate patterns, 3.6b discovery descriptor completeness, 3.6c parents vs dependsOn consistency). All findings feed Category 1 (Manifest Design).
• review-core-plugin.md — Step 5.6 with the same checks scoped to changed manifest files only.
• create-core-plugin.md — Step 4e.1 post-scaffold enrichment that walks each user-supplied field to add a discovery block and gates any scaffolding.rules against 5 explicit reject patterns.

No changes to plugin-best-practices.md or plugin-review-guidance.md — the additions are procedural workflow steps, not narrative rules.

Track B — consumer skill (cross-repo)

The databricks-apps skill in databricks/databricks-agent-skills#79 ("docs(apps): update AppKit scaffolding setup for Manifest 2.0") adds a new "Scaffolding Rules Protocol" subsection to skills/databricks-apps/SKILL.md (skill version bump 0.1.1 → 0.1.2). It consumes the rules emitted by this PR:

1. Gather rules from selected --features plugins + every plugin with requiredByTemplate: true + template-level
2. Precedence — manifest rules override skill-baked directives
3. Phase ordering — Before init / After init prefixes
4. Conflict detection — plugin must ↔ template never stops the flow
5. Reporting — surface merged guardrails to the user before init

Sequencing: Track A (this PR) ships first. Track B is opened, not blocking merge — plugin-level rules are advisory metadata in the transition state. Failure mode if Track B lands later: degraded (third-party plugin guardrails don't fire), not broken (resources still drive the build).

End-to-end integration runbook lives in ~/.xavier/research/manifest-2-skill-integration-test.md (vault note, not in this repo).

Deferred to follow-up PRs

Change	Note
TypeScript-authored manifests (defineManifest({ ... }))	Future work; no stacked PR yet
Docs for the v2.0 format	#370 (open, branch manifest-docs)
Branded field-name keys for compile-time dependsOn checks	Not a must-have, larger implementation change
cli variant full hardening	Shell-metacharacter denylist + describe notes shipped here. Remaining: argv-array form (executor-side spawn-not-shell), output-shape contract (replace jq path with declared field names + unwrap). Tied to the executor PR.
Track B skill PR	databricks-agent-skills#79 — opens for skill author review; not required to merge before Track A

Note

Plugin authors continue to write manifest.json. See the docs PR (#370) for the authoring guide.

Note on the generated files diff

1. Zod emits keys in a different order than the hand-written JSON Schema did. For example, you'll see "description" flip from before "type" to after "type", and "required": [...] arrays appear in different positions. z.toJSONSchema() has its own key ordering, and that's what now writes the file.
2. Discovery is reshaped as a discriminated union. That's why $ref: "#/$defs/resourceRequirement" got expanded into giant oneOf: [...] blocks per resource type — the schema's actual shape changed, not just its layout.

Biome's JSON formatter only normalizes whitespace; it doesn't reorder keys. So the noise you're seeing is the cost of moving from hand-authored JSON Schema → Zod-generated JSON.

[Copilot]

Pull request overview

This PR upgrades the AppKit plugin manifest ecosystem to a v2.0 template manifest format to support smarter databricks apps init scaffolding, including field discovery metadata, post-scaffold user instructions, computed field origins, and semantic (cross-field) validation during plugin validate.

Changes:

• Bump template plugin manifest version to 2.0 and add a top-level scaffolding descriptor.
• Extend plugin/template schemas with discovery (CLI command-based value discovery) and postScaffold steps; generate/propagate computed origin into template manifests during sync.
• Add semantic validation (dependsOn cycles/dangling refs, <PROFILE> placeholder, discovery/origin coherence, postScaffold structure) and associated tests/docs updates.

Reviewed changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
template/appkit.plugins.json	Updates template manifest to v2.0 and annotates built-in plugins with discovery/origin/postScaffold plus scaffolding descriptor.
packages/shared/src/schemas/template-plugins.schema.json	Expands template manifest schema to support v2.0 and scaffolding descriptor; inlines field/requirement defs for origin.
packages/shared/src/schemas/plugin-manifest.schema.json	Adds discovery to resource fields and postScaffold to plugin manifests.
packages/shared/src/schemas/plugin-manifest.generated.ts	Updates generated TS types to include discovery/postScaffold types.
packages/shared/src/plugin.ts	Re-exports new generated types (DiscoveryDescriptor, PostScaffoldStep).
packages/shared/src/cli/commands/plugin/validate/validate.ts	Runs semantic validation and formats semantic errors/warnings in CLI output.
packages/shared/src/cli/commands/plugin/validate/validate-manifest.ts	Implements semantic validation rules and issue formatting.
packages/shared/src/cli/commands/plugin/validate/validate-manifest.test.ts	Adds test coverage for new semantic validation behavior.
packages/shared/src/cli/commands/plugin/sync/sync.ts	Computes/injects origin, bumps template manifest to v2.0, and adds scaffolding descriptor on write.
packages/shared/src/cli/commands/plugin/sync/sync.test.ts	Adds unit tests for origin computation.
packages/shared/src/cli/commands/plugin/manifest-types.ts	Adds scaffolding descriptor types and exports new manifest-related types.
packages/appkit/src/plugins/lakebase/manifest.json	Adds discovery descriptors and postScaffold steps to the lakebase built-in plugin manifest.
packages/appkit/src/plugins/genie/manifest.json	Adds schema reference, discovery descriptor, and postScaffold steps to genie plugin manifest.
packages/appkit/src/plugins/files/manifest.json	Adds discovery descriptor and postScaffold steps to files plugin manifest.
packages/appkit/src/plugins/analytics/manifest.json	Adds discovery descriptor and postScaffold steps to analytics plugin manifest.
docs/static/schemas/template-plugins.schema.json	Publishes updated template plugins schema for docs site.
docs/static/schemas/plugin-manifest.schema.json	Publishes updated plugin manifest schema for docs site.
docs/static/appkit-ui/styles.gen.css	Updates generated UI styles (tailwind output) used by docs UI.
docs/docs/api/appkit/Interface.ResourceFieldEntry.md	Documents the new discovery field on ResourceFieldEntry.
docs/docs/api/appkit/Interface.PluginManifest.md	Documents the new postScaffold field on PluginManifest.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[keugenek]

@atilafassina this one looks ok and evolution, however little heavy code-wise (let me post comments from isaac about these) and mixing concepts little bit. What do you think of this:

Alternatives worth weighing

Zod as single source of truth (my default for a TS-first ecosystem):

• Types, runtime validation, and semantic checks in one file.
• zod-to-json-schema emits the $schema JSON for VSCode intellisense.
• Cycle/dangling-ref checks become .refine() calls in the same module as the
  schema.
• Deletes plugin-manifest.generated.ts and the Ajv layer.

TS-authored manifests compiled to JSON:

• manifest.ts → defineManifest({...}) with branded types for field names
  (compile-time dependsOn checking, no DFS needed).
• sync lowers to .json for non-TS consumers.
• Strongest author ergonomics; agent still reads JSON.

Split concerns into two files:

• manifest.json — strict contract (resources, fields, env), stable schema.
• scaffold.yaml or .md — agent hints (discovery, prompts, postScaffold), free
  to evolve.
• Today they're fused, which is why the schema keeps growing.

[keugenek]

Overall direction looks right — v2.0 versioning via if/then is correct and co-located manifest.json per plugin is the right locality. A few shape-level concerns worth addressing before or soon after this lands. Happy to chat on any of them.

[MarioCadenas]

how does the agent know how to do this? should we add some command?

[atilafassina]

It has just used the CLI (databricks apps manifest) so it's right in their context to find the command. In my xp it's capable of that.

I'm torn between adding a CLI command in the manifest to make it more effectivebecause such a contract is not enforced anywhere. I'm assuming that CLI commands are sufficiently self discoverable.

[MarioCadenas]

same as the comment before, how does it know how to do it?

[atilafassina]

It has just used the CLI (databricks apps manifest) so it's right in their context to find the command. In my xp it's capable of that.

I'm torn between adding a CLI command in the manifest to make it more effectivebecause such a contract is not enforced anywhere. I'm assuming that CLI commands are sufficiently self discoverable.

[MarioCadenas]

do we need to bump this to 2.0? is just incremental no? no breaking changes

[atilafassina]

you're right if we only look at the schema, but the validation treats the some of the new props as required, so version 1.1 will fail 2.0 validation.

If we bump to 1.2 the strict way would be to make the validation looser.

[MarioCadenas]

these 2 flags are not required as far as I know no?

[MarioCadenas]

1.2 maybe?

[atilafassina]

same as above #261 (comment)

we can, but validation would need to be less strict.

[MarioCadenas]

why is this one removed?

[atilafassina]

I think this was an accident when I split docs PR from this one. Let me revert it.

[MarioCadenas]

I don't think the template has this command in the scripts in package.json no? also it doesn't use pnpm

[atilafassina]

replacing this wiht a package and ORM agnostic rule.

"run migrations". I'll also downgrade it to a should