ES→OS: endpoint to audit and rewrite VTL that calls the deprecated $estool.esSearch / $estool.esRaw

[fabrizzio-dotCMS]

Description

Customers migrating from Elasticsearch to OpenSearch 3 have to find and rewrite every Velocity
template that calls the two deprecated search methods, $estool.esSearch(...) and
$estool.esRaw(...). Today this is entirely manual, and it is the single hardest part of the
migration for a customer to do alone:

• The code is spread across four places — templates and containers stored in the database, file
  templates/containers in the assets tree, widget code and custom-field Velocity inside content type
  definitions, and Velocity script steps inside workflow actions. Nothing but the files can be
  grepped.
• The failure is silent. These templates keep working through Phase 1 (reads still come from
  Elasticsearch) and break in Phase 2, and they break by rendering less, not by erroring.

This task adds one role-gated endpoint that both reports what needs changing and, when explicitly
instructed, applies the mechanical part of the change with a manifest that can be reverted.

Endpoint modes

Mode	Behaviour
GET /api/v1/index/migration/vtl-audit	Read-only report: object type, id, name, line, rule, confidence. Touches nothing.
POST /api/v1/index/migration/vtl-audit/_apply with dryRun: true	Returns the exact diff it would write. Writes nothing.
POST /api/v1/index/migration/vtl-audit/_apply	Writes the working version only of each object — never publishes — and returns a runId plus the manifest (object, rule applied, previous version).
POST /api/v1/index/migration/vtl-audit/_revert with a runId	Restores the previous version of every object in that manifest.

Rules

Auto-rewritable (high confidence):

From	To
$estool.esRaw(	$estool.raw(
$r.took (on a variable bound to a raw response)	$r.tookInMillis
$r.hits.getAt(n)	$r.hits.hits.get(n)
$estool.esSearch( and the $item.map.field uses of its loop variable	$estool.search( and $item.field

Reported, never rewritten:

Found	Why it needs a human
$r.toString() used as JSON	The neutral response does not serialize to engine JSON; the JSON has to be rebuilt field by field. Highest-risk finding: it fails silently.
$results.hits.maxScore	No equivalent on the neutral response; the score has to be read per hit.
esSearch whose result variable escapes the file (#parse, #dotParse, a macro)	The rewrite of .map. uses cannot be attributed safely.
Velocity steps in workflow actions	Reported with the action id; rewriting is allowed only with an explicit target list.

Design constraints

1. Only rewrite what is provably local. If the result variable of an esSearch call crosses a
   #parse / #dotParse / macro boundary, the finding is reported, not rewritten. A wrong rewrite
   here fails the same silent way the migration itself does.
2. Never publish. Writing the working version keeps the live site untouched and uses the
   versioning that templates, containers, content types and file assets already have — restoring the
   previous version is the rollback. Publishing stays a human decision after reading the diff.
3. The manifest is the only backup we own. Workflow actionlets do not version, so for those the
   previous code is stored whole in the manifest.
4. Idempotent. A second run finds nothing to do; _revert works as long as the runId manifest
   exists.
5. Gated and off by default. Same gate as the readiness endpoint — CMS Administrator plus the
   migration role — since this reads and rewrites customer code.

Acceptance Criteria

• GET .../vtl-audit returns findings from all four sources: database templates and containers,
  file templates/containers, content type field values (widget code and custom fields), and
  Velocity workflow actionlets.
• Every finding carries object type, id, human-readable name, line, rule, and whether it is
  auto-rewritable or manual.
• _apply with dryRun: true returns a diff and provably writes nothing.
• _apply writes only working versions; nothing is published, and the live site renders
  identically immediately after a run.
• _apply returns a runId whose manifest names every object touched and its previous version.
• _revert with that runId restores every object to its previous state, including workflow
  actionlets, which have no versions of their own.
• Re-running _apply on an already-converted installation reports and changes nothing.
• A template whose esSearch result escapes through #parse or a macro is reported and left
  untouched.
• toString()-as-JSON and maxScore findings are never rewritten.
• Both endpoints answer 403 without CMS Administrator plus the migration role.
• Integration tests cover a fixture with one case per rule, plus an already-migrated fixture that
  must produce zero findings.

Additional Context

The rules come from the customer-facing migration guide's template section, which was verified
against the code:

• ESContentTool.esSearch returns ESSearchResults holding raw Contentlets; ESContentTool.search
  returns ContentSearchResults<ContentMap>. ContentMap has no getMap(), which is why
  $item.map.field stops resolving after the rename — silently.
• ContentSearchResponse exposes getTookInMillis(), and SearchHits exposes getHits() /
  getTotalHits(); the old getAt(n) and getMaxScore() have no counterpart.
• On Elasticsearch the neutral search() delegates to the same esSearch() underneath
  (ESSearchAPIImpl.search), so query handling is unchanged by the rewrite — only the result type is.

Both deprecated methods are marked @Deprecated(forRemoval = true) for removal in v26.08.04.

Related: the deprecation inventory in docs/backend/ES_OS_DEPRECATIONS.md and the plugin audit skill
requested in #37466 — this issue is the Velocity half, and the two should share one rule source
rather than each keeping its own list.

[fabrizzio-dotCMS]

Where the code actually lives — verified against the schema and the code

Findings from verifying the customer migration guide. These are the four stores the audit has to read
and the apply/revert modes have to write, with the exact locations.

1. Files, held as file assets inside dotCMS

What	Path inside the site
File template	/application/templates/<name>/body.vtl
File container	/application/containers/<name>/container.vtl, plus one .vtl per content type it serves
Theme shell	template.vtl
Scripting endpoints	/application/apivtl/<folder>/{get,post,put,delete}.vtl — served as an API by VTLResource (VTL_PATH = "/application/apivtl"). Easy to forget, and the likeliest home of esRaw(...).toString()

Those readable paths exist only inside dotCMS: on disk each file asset sits under the assets
directory in <assets>/<inode[0]>/<inode[1]>/<inode>/ (FileUtil.binaryPath), so a filesystem sweep
finds text but cannot name what it found. The audit should report the CMS path, not the disk path.

The telemetry collectors confirm the conventions — they query
parent_path LIKE '/application/templates/%' AND asset_name = 'body.vtl' and the equivalent for
container.vtl.

2. The database — templates and containers created in the admin UI

Seven columns across three tables can hold Velocity:

Table	Columns
template	body, header, footer, head_code
dot_containers	code, pre_loop, post_loop
container_structures	code — one row per content type the container serves

(From postgres.sql: template at line 682, container_structures at 750, dot_containers at
1223.) Two neighbours to skip explicitly: template.drawed_body is the visual editor's layout JSON,
and dot_containers.lucene_query is a query, not code.

Note the table name is dot_containers, not container.

An audit that only looks at template.body misses three columns per template.

3. Content type definitions

Widget code and the Velocity inside custom fields live in the field definition, not in any template.
A widget is frequently a single #dotParse('/application/vtl/…'), so a hit there is often just an
include and the real code is in the file it names — the audit should follow that one hop.

4. Workflow actions

The code of a Velocity script step (VelocityScriptActionlet) lives in the action's parameters.
No file, no template, and nothing greppable reaches it.

What this implies for the tool

Recoverability is not uniform, and this decides how _apply and _revert must behave:

Store	Versioned?	Consequence
Templates, containers	Yes (TemplateAPI.findAllVersions)	Writing a new working version is the backup; revert = restore previous version
File assets	Yes, as file asset versions	Same
Content type fields (widget code, custom fields)	No	Saving destroys the previous code. The manifest must store it whole
Workflow action parameters	No	Same

So the manifest is not optional bookkeeping for stores 3 and 4 — it is the only copy that will exist.

Cache: writing through the APIs invalidates for us (TemplateAPIImpl:443 removes the template
cache entry, TemplateLoader.invalidate regenerates the Velocity file). A tool that ever writes a
file asset straight to disk would bypass that and leave pages rendering the old code — another reason
_apply should go through the APIs only.

Reporting: every finding should carry the store, the CMS-visible name/path (never the internal
disk path), the column or field it came from, and the line — otherwise a customer cannot act on the
report without a DBA.

[fabrizzio-dotCMS]

The rules in this issue come from the template section of the customer-facing ES→OpenSearch 3 migration guide, where each one is shown as a before/after pair with the reasoning: https://claude.ai/code/artifact/20d61d95-bebf-4f0d-9911-2796f0ac99a8

Useful as the spec for what the tool has to detect and rewrite — in particular the split between what is mechanically rewritable (the method renames, took → tookInMillis, getAt(n)) and what can only be reported (toString() used as JSON, maxScore, and any esSearch whose result variable escapes through #parse or a macro).

[github-actions]

PRs linked to this issue

• docs(search-api): fix the VTL examples and document JSON output by @fabrizzio-dotCMS