Implement search and versioned Prim docs for the new Pursuit

[thomashoneyman]

The new Pursuit application in #789 will serve package and module documentation mostly as static HTML, but we still need a server for search. We also need to replace the way Pursuit currently gets documentation for Prim. (Content below is written with Sol assistance).

Search

Current Pursuit builds an in-memory search index from the latest version of every package. It supports searching for packages, modules, declarations, and approximate type signatures, and uses reverse dependencies as part of its ranking.

We have another PureScript implementation of much of this functionality in Spago's docs-search. We should move or adapt the reusable parts of that implementation into registry-dev, in particular:

• the type query parser;
• normalization of types into searchable shapes;
• approximate type matching and ranking;
• reconstruction of signatures for constructors and type class members.

We should adapt this code to use the compiler-independent docs types from #704 rather than the compiler's internal type representation.

Spago's storage and loading model is intended for a local static documentation site, so we should not preserve the JavaScript index shards or browser-side tries. Instead, we can build an index in SQLite as packages are processed by the new Pursuit.

At minimum the index needs to represent:

• packages and package versions;
• modules;
• declarations and child declarations;
• normalized type signatures and searchable type shapes;
• package popularity or reverse-dependency counts;
• the compiler versions supported by each package version.

The docs JSON remains the source of truth. The SQLite index is derived data and should be fully rebuildable. Processing the same package version again should replace its rows transactionally, and unpublishing a version should remove them.

Normal search should probably continue to use only the latest version of each package, as Pursuit does today, while keeping older versions available through their direct documentation URLs. We should preserve the useful parts of the current search behavior, including approximate type matching, package popularity in ranking, result URLs and fragments, and the existing JSON/CORS behavior used by clients.

One reason compiler compatibility was added to registry metadata in #669 was to eventually support filtering Pursuit search results by compiler version. Since we now have that information, the new index and API should support filtering by a compiler version or range without having to regenerate declaration rows whenever a package's compiler matrix changes.

We should compare representative queries against both current Pursuit and Spago's docs search before switching production over. The algorithms do not have to be identical, but surprising changes in result quality should be intentional.

Prim documentation

Pursuit currently gets Prim documentation from the PureScript compiler library it was built with. That means the available builtins implicitly change when Pursuit updates its compiler dependency, and there is no way to view the Prim docs associated with an older compiler.

Instead, we should materialize Prim docs for each compiler version supported by the registry and store them alongside ordinary package docs.

When the registry discovers a compiler for which there is no existing Prim artifact, it can enqueue a job which invokes that compiler on the minimal project necessary to produce its builtin docs. The result should use the same canonical docs format as packages where possible, be uploaded under the compiler version, and be included in the search index.

We should add versioned builtins URLs while preserving the existing unversioned /builtins/docs/... URLs as aliases to whichever compiler version Pursuit considers current.

The set of available compiler binaries should be reconciled against the set of stored Prim artifacts so that a missed or failed job can be repaired. We should not rely only on the existing "new compiler missing from prelude metadata" check, since that is a proxy used to start compiler matrix work rather than a durable record of which Prim docs exist.

This issue can be closed when the new Pursuit supports package, module, declaration, and approximate type search from its SQLite index; compiler filtering works; and versioned Prim docs are generated, searchable, and served through both versioned and compatibility URLs.

It depends on the canonical docs format in #704 and is required before the new Pursuit application can replace production Pursuit.