diff --git a/docs/backend/OPENSEARCH_MIGRATION.md b/docs/backend/OPENSEARCH_MIGRATION.md index daf868a7b737..756032480f88 100644 --- a/docs/backend/OPENSEARCH_MIGRATION.md +++ b/docs/backend/OPENSEARCH_MIGRATION.md @@ -108,6 +108,46 @@ In Phase 2 OS serves reads but ES is still active. If OS throws an exception on `PhaseRouter` catches it, logs at **ERROR** level, and retries against ES automatically. - The caller receives a correct result from ES + +The fallback log line names the operation and the root cause, so an outage is identifiable from +logs alone: + +``` +OS read failed in Phase 2 [indexCount] — falling back to ES. OS index may be stale or +unavailable. Cause: An error occurred when executing the Lucene Query +/ root cause: ConnectException: Connection refused +``` + +**Content search reached this fallback only from #37413 onward.** Before that fix, +`ESContentFactoryImpl` selected a provider with a bare ternary and called it directly from its +five read call sites, so the router — and therefore the fallback — was unreachable from the +busiest read path in the product. An OpenSearch outage in Phase 2 returned a well-formed `200` +with zero results for content types whose count was already cached, and a `500` for the rest. +The empty result is the dangerous variant: a caller cannot tell it apart from "this content type +has no content". If you are reading this against a build that predates that fix, the guarantee +above did not hold for `/api/content/_search`, `ContentletAPI` search and count, Velocity +`$dotcontent.pull`, URL maps, Site Search or the admin content browser. + +Two conditions are required for the fallback to fire, and both took work: + +1. **The read must go through `PhaseRouter`.** Provider selection in `ESContentFactoryImpl` now + has exactly one mechanism; a second one is what caused #37413. +2. **The OpenSearch provider must actually raise.** In Phase 2, `ContentFactoryIndexOperationsOS` + raises instead of absorbing a failure into a legitimate-looking success — the `ERROR_HIT` + sentinel, `-1` for a count, an empty scroll list, and the index-resolution guard in + `searchHits` that returned an empty result for *any* exception. A provider that reports + success gives the router nothing to catch. Outside Phase 2 the absorbing behaviour is + unchanged, and `ContentFactoryIndexOperationsES` is not touched at all. + +**What still does not fall back**, by design or by limitation: + +| Situation | Behaviour | Why | +|---|---|---| +| Phase 3 read failure | Propagates | ES is decommissioned; a silent fallback would report data that is genuinely absent | +| Failure mid-scroll | Propagates | An OS scroll id is meaningless to ES, so a half-drained scroll cannot be resumed on the other engine. `createScrollQuery` is routed for provider selection only | +| OS answers successfully with stale or incomplete data | No fallback | Indistinguishable from a query that legitimately matches nothing; falling back on every empty result would double the load of most searches | +| A repeated identical query during an outage | Served from the query cache | The cached value is a real earlier result, so serving it is correct — but it makes an outage invisible when reproducing by hand. Vary `offset` per call. The error *sentinel* is never cached on the Phase 2 path, so a poisoned entry cannot outlive the outage | +| Any phase, read failure reaching the legacy layer | Empty `200` | `ContentUtils`' `catch (Throwable)` and `ContentHelper`'s `resultsSize = 0` overwrite predate the migration and affect pure-ES installs too. Tracked separately; this is why the outage was silent rather than loud | - The ERROR log makes the OS failure visible for operators - In Phase 3 there is no fallback — ES is decommissioned and OS failures propagate normally diff --git a/docs/backend/OPENSEARCH_MIGRATION_TEST_PLAN.md b/docs/backend/OPENSEARCH_MIGRATION_TEST_PLAN.md index fdd790dabf07..299f0e5dec0c 100644 --- a/docs/backend/OPENSEARCH_MIGRATION_TEST_PLAN.md +++ b/docs/backend/OPENSEARCH_MIGRATION_TEST_PLAN.md @@ -834,13 +834,27 @@ When the migration starts **successfully** (no shutdown) you instead see an `INF - **Steps:** 1. Confirm content is searchable in Phase 2. 2. Break OpenSearch reads (stop OS or remove the OS working index). - 3. Run the same search again. + 3. Search again — but **change the query on every call** (see the warning below). + 4. Also search a content type you have **not** queried since the server started, so its count is + not cached either. That variant used to fail differently: a `500` instead of an empty `200`. - **Expected Result:** - - The search still returns the correct result (served from ES). - - The log shows an ERROR similar to - `OS read failed in Phase 2 — falling back to ES. OS index may be stale or unavailable. Cause: …` + - The search still returns the correct result (served from ES), for every content type that has + live content. Not zero results, and not a `500`. + - The log shows an ERROR naming the operation and the root cause, similar to + `OS read failed in Phase 2 [indexCount] — falling back to ES. OS index may be stale or + unavailable. Cause: … / root cause: ConnectException: Connection refused` - **Type:** Manual +> ⚠️ **This case gave a false PASS before #37413 and can do so again.** Repeating an *identical* +> search returns the pre-outage result from the query cache, which is indistinguishable from a +> working fallback. Vary the query — a different `offset` on each call is enough — or you are +> testing the cache, not the fallback. A count is cached without `offset` in its key, so also +> exercise a content type never queried since startup. +> +> A second false-pass route: if a *count* is what fails, some paths propagate the error while the +> *search* path used to convert it to an empty result on the spot. Check the returned total, not +> just the absence of an error. + ## TC-040 — Phase 3 does NOT auto-rollback (negative case) - **Objective:** Document that the automatic fallback to ES exists only in Phases 1–2. In Phase 3 (ES diff --git a/dotCMS/src/main/java/com/dotcms/content/elasticsearch/business/ESContentFactoryImpl.java b/dotCMS/src/main/java/com/dotcms/content/elasticsearch/business/ESContentFactoryImpl.java index 8b3a7a286d9a..6dfbf9720eb4 100644 --- a/dotCMS/src/main/java/com/dotcms/content/elasticsearch/business/ESContentFactoryImpl.java +++ b/dotCMS/src/main/java/com/dotcms/content/elasticsearch/business/ESContentFactoryImpl.java @@ -1,8 +1,6 @@ package com.dotcms.content.elasticsearch.business; import static com.dotcms.content.elasticsearch.business.ESContentletAPIImpl.MAX_LIMIT; -import static com.dotcms.content.index.IndexConfigHelper.isMigrationComplete; -import static com.dotcms.content.index.IndexConfigHelper.isReadEnabled; import static com.dotcms.variant.VariantAPI.DEFAULT_VARIANT; import static com.dotmarketing.portlets.contentlet.model.Contentlet.AUTO_ASSIGN_WORKFLOW; import static com.dotmarketing.portlets.contentlet.model.Contentlet.TITLE_IMAGE_KEY; @@ -17,6 +15,7 @@ import com.dotcms.content.business.json.ContentletJsonAPI; import com.dotcms.content.business.json.ContentletJsonHelper; import com.dotcms.content.index.ContentFactoryIndexOperations; +import com.dotcms.content.index.PhaseRouter; import com.dotcms.content.index.IndexContentletScroll; import com.dotcms.content.index.domain.SearchHit; import com.dotcms.content.index.domain.SearchHits; @@ -232,8 +231,19 @@ public class ESContentFactoryImpl implements ContentletFactory { private final ContentletCache contentletCache; private final LanguageAPI languageAPI; - private final ContentFactoryIndexOperations indexOperationsES; - private final ContentFactoryIndexOperations indexOperationsOS; + /** + * Phase-aware router for every index read this factory performs. + * + *

These read call sites are the funnel for essentially all content search in the product: + * {@code /api/content/_search}, {@code ContentletAPI} search and count, Velocity + * {@code $dotcontent.pull}, URL maps, Site Search, scroll consumers and the admin content + * browser. They used to pick a provider with a bare ternary and call it directly, which left + * the router's Phase 2 fallback to Elasticsearch unreachable from the busiest read path in + * the product — with OpenSearch down, a search returned a well-formed empty result rather + * than the data Elasticsearch was holding the whole time (issue #37413). The router is now + * the only way a provider is selected here; do not reintroduce a second mechanism.

+ */ + private final PhaseRouter indexRouter; private static final ObjectMapper mapper = DotObjectMapperProvider.getInstance() .getDefaultObjectMapper(); @@ -260,18 +270,28 @@ public String getInode() { * Elastic index. */ public ESContentFactoryImpl() { - this.contentletCache = CacheLocator.getContentletCache(); - this.languageAPI = APILocator.getLanguageAPI(); - this.indexOperationsOS = new ContentFactoryIndexOperationsOS(); - this.indexOperationsES = new ContentFactoryIndexOperationsES(CacheLocator.getESQueryCache()); + this(new ContentFactoryIndexOperationsES(CacheLocator.getESQueryCache()), + new ContentFactoryIndexOperationsOS()); } /** - * Migration-phase-aware Operations delegate - * @return {@link ContentFactoryIndexOperations} + * Constructor that accepts both index-operation providers, so a test can substitute one of + * them — typically an OpenSearch provider that fails on every read, to exercise the Phase 2 + * fallback to Elasticsearch. + * + *

This exists only for testing: production code must use {@link #ESContentFactoryImpl()}, + * which supplies the real providers. It carries no behaviour of its own — provider selection + * for reads is decided by {@link com.dotcms.content.index.PhaseRouter}, not here.

+ * + * @param indexOperationsES the Elasticsearch provider + * @param indexOperationsOS the OpenSearch provider */ - ContentFactoryIndexOperations indexOperationsDelegate(){ - return isMigrationComplete() || isReadEnabled() ? indexOperationsOS : indexOperationsES ; + @VisibleForTesting + ESContentFactoryImpl(final ContentFactoryIndexOperations indexOperationsES, + final ContentFactoryIndexOperations indexOperationsOS) { + this.contentletCache = CacheLocator.getContentletCache(); + this.languageAPI = APILocator.getLanguageAPI(); + this.indexRouter = new PhaseRouter<>(indexOperationsES, indexOperationsOS); } @Override @@ -1349,7 +1369,8 @@ public List findContentlets(final List inodes) throws DotDat public List findContentletsByHost(final String hostId, final int limit, final int offset) { try { - final List inodes = indexOperationsDelegate().search("+conhost:" + hostId, limit, offset); + final List inodes = indexRouter.read("search", + impl -> impl.search("+conhost:" + hostId, limit, offset)); return findContentlets(inodes); } catch (Exception e) { throw new RuntimeException(e.getMessage(), e); @@ -1604,7 +1625,7 @@ public List getRelatedLinks(Contentlet contentlet) throws DotDataException public long indexCount(final String query) { final String qq = LuceneQueryDateTimeFormatter .findAndReplaceQueryDates(translateQuery(query, null).getQuery()); - return indexOperationsDelegate().indexCount(qq); + return indexRouter.read("indexCount", impl -> impl.indexCount(qq)); } @Override @@ -1613,8 +1634,8 @@ public SearchHits indexSearch(final String query, final int limit, final int off final String formattedQuery = LuceneQueryDateTimeFormatter .findAndReplaceQueryDates(translateQuery(query, sortBy).getQuery()); - return indexOperationsDelegate().searchHits( - formattedQuery, limit, offset, sortBy); + return indexRouter.read("searchHits", + impl -> impl.searchHits(formattedQuery, limit, offset, sortBy)); } @@ -1631,7 +1652,8 @@ public SearchHits indexSearch(final String query, final int limit, final int off * @return PaginatedArrayList containing all search results */ PaginatedArrayList indexSearchScroll(final String query, String sortBy) { - return indexOperationsDelegate().indexSearchScroll(query, sortBy, SCROLL_BATCH_SIZE.get()); + return indexRouter.read("indexSearchScroll", + impl -> impl.indexSearchScroll(query, sortBy, SCROLL_BATCH_SIZE.get())); } /** @@ -1666,7 +1688,15 @@ PaginatedArrayList indexSearchScroll(final String query, Strin public IndexContentletScroll createScrollQuery(final String luceneQuery, final User user, final boolean respectFrontendRoles, final int batchSize, final String sortBy) { - return indexOperationsDelegate().createScrollQuery(luceneQuery, user, respectFrontendRoles, batchSize, sortBy); + // Routed for provider selection only: the fallback here can never fire, and that is + // correct rather than an oversight. Both provider implementations just construct a + // cursor -- no I/O, nothing to throw -- and the requests happen later, inside that + // cursor, outside the router. A mid-scroll fallback is impossible in principle anyway: + // an OpenSearch scroll id is meaningless to Elasticsearch, so a half-drained scroll + // cannot be resumed on the other engine. Residual gap: a consumer already iterating a + // scroll when OpenSearch dies still fails (issue #37413, AC-002 documented exclusion). + return indexRouter.read("createScrollQuery", impl -> impl.createScrollQuery( + luceneQuery, user, respectFrontendRoles, batchSize, sortBy)); } /** diff --git a/dotCMS/src/main/java/com/dotcms/content/index/PhaseRouter.java b/dotCMS/src/main/java/com/dotcms/content/index/PhaseRouter.java index 03754b5d5a7a..eaf71ab42287 100644 --- a/dotCMS/src/main/java/com/dotcms/content/index/PhaseRouter.java +++ b/dotCMS/src/main/java/com/dotcms/content/index/PhaseRouter.java @@ -185,19 +185,76 @@ public List writeProviders() { * @return result from the read provider (or ES fallback in Phase 2) */ public R read(final Function fn) { + return read(null, fn); + } + + /** + * Same as {@link #read(Function)}, but names the operation in the fallback log line. + * + *

The plain {@link #read(Function)} can only report the cause, because the operation it + * runs is an opaque lambda. That is enough to know something fell back, but not + * enough for log-based monitoring to say what stopped working — which is the + * early-warning signal the migration design promises. Callers on a read path that matters + * operationally should pass a name.

+ * + * @param operation short name of the operation being routed, e.g. {@code "indexCount"}; + * may be {@code null}, in which case the log line omits it + * @param fn must not throw checked exceptions; use {@link #readChecked} otherwise + */ + public R read(final String operation, final Function fn) { if (!isPhase2()) { return fn.apply(readProvider()); } try { return fn.apply(osImpl); } catch (final RuntimeException e) { - Logger.error(PhaseRouter.class, - "OS read failed in Phase 2 — falling back to ES. " - + "OS index may be stale or unavailable. Cause: " + e.getMessage(), e); + Logger.error(PhaseRouter.class, fallbackMessage(operation, e), e); return fn.apply(esImpl); } } + /** + * Builds the Phase 2 fallback log line. + * + *

Named operation and root cause are both deliberate. This line is the migration + * design's early-warning signal that OS has stopped answering, so it has to say + * what stopped working and why — a monitor that can only see "a read fell + * back" cannot tell a dead node from a single malformed query. The root cause matters + * because providers wrap failures in a generic message: the actionable text + * ("Connection refused", "index_not_found_exception") is further down the chain.

+ */ + private static String fallbackMessage(final String operation, final Throwable failure) { + final StringBuilder message = new StringBuilder("OS read failed in Phase 2"); + if (null != operation && !operation.isBlank()) { + message.append(" [").append(operation).append(']'); + } + message.append(" — falling back to ES. OS index may be stale or unavailable. Cause: ") + .append(failure.getMessage()); + final String rootCause = rootCauseMessage(failure); + if (null != rootCause) { + message.append(" / root cause: ").append(rootCause); + } + return message.toString(); + } + + /** + * Message of the deepest cause, or {@code null} when the failure is not wrapping anything + * that adds information. + */ + private static String rootCauseMessage(final Throwable failure) { + Throwable current = failure; + while (null != current.getCause() && current.getCause() != current) { + current = current.getCause(); + } + if (current == failure) { + return null; + } + final String message = current.getMessage(); + return null == message || message.isBlank() + ? current.getClass().getSimpleName() + : current.getClass().getSimpleName() + ": " + message; + } + /** * Fans a void write out to all current write providers. * diff --git a/dotCMS/src/main/java/com/dotcms/content/index/opensearch/ContentFactoryIndexOperationsOS.java b/dotCMS/src/main/java/com/dotcms/content/index/opensearch/ContentFactoryIndexOperationsOS.java index 8cd2832390d0..9347793c286e 100644 --- a/dotCMS/src/main/java/com/dotcms/content/index/opensearch/ContentFactoryIndexOperationsOS.java +++ b/dotCMS/src/main/java/com/dotcms/content/index/opensearch/ContentFactoryIndexOperationsOS.java @@ -84,6 +84,33 @@ private boolean shouldQueryCache(final String exceptionMsg) { exception.contains("search_phase_execution_exception"); } + + /** + * Whether a failure this provider would otherwise absorb must be raised instead. + * + *

These read paths historically convert a failure into a legitimate-looking empty result + * — an empty {@code SearchHits}, the {@code ERROR_HIT} sentinel, {@code -1} for a count, an + * empty scroll list. The provider then reports success, so the phase router sees nothing to + * catch and cannot fall back: with OpenSearch unable to answer in Phase 2, a search returned + * zero results while Elasticsearch held the data the whole time (issue #37413).

+ * + *

The change is scoped to Phase 2 on purpose, and the scoping is what makes it safe. In + * Phase 2 the router catches the raised failure immediately above this class and turns it + * into a successful Elasticsearch read, so no caller ever observes a new exception + * type — which is why no enumeration of callers relying on empty-instead-of-throw + * is needed. In every other phase the absorbing behaviour is untouched: phases 0 and 1 do + * not read from OpenSearch at all, and Phase 3 has no Elasticsearch to fall back to, so + * raising there would expose every such caller with nothing gained.

+ * + *

The one genuine behaviour change: when both engines fail on the same read the caller now + * receives an error rather than a silent empty result. That is the improvement being asked + * for — the silent variant is the dangerous one, because a caller cannot tell it apart from + * "this content type has no content".

+ */ + private static boolean mustRaiseForPhase2Fallback() { + return IndexConfigHelper.isReadEnabled() && !IndexConfigHelper.isMigrationComplete(); + } + /** * If enabled SearchRequests are executed and then cached */ @@ -115,6 +142,21 @@ private HitsMetadata cachedIndexSearch(final SearchRequest searchRequest Logger.warn(this.getClass(), String.format("OS Query: %s", String.valueOf(searchRequest))); Logger.warn(this.getClass(), String.format("Class %s: %s", e.getClass().getName(), exceptionMsg)); Logger.warn(this.getClass(), "----------------------------------------------"); + if (mustRaiseForPhase2Fallback()) { + // Not cached: a sentinel stored here would be replayed to every later identical + // query as a successful empty result, outliving the outage and defeating the + // fallback even after OpenSearch recovers. + // Index name and OpenSearch's own reason only. The full SearchRequest is + // deliberately left out: in Phase 2 the router logs this message at ERROR, and a + // Lucene query can carry end-user search terms and field values (Constitution + // Principle III). The request body is still available at DEBUG on the WARN block + // above for anyone diagnosing a specific query. + throw new DotRuntimeException(String.format( + "OpenSearch search failed on index [%s]: %s", + (searchRequest.index() != null) ? String.join(",", searchRequest.index()) + : "unknown", + exceptionMsg), e); + } if(shouldQueryCache(exceptionMsg)) { queryCache.put(searchRequest, ERROR_HIT); } @@ -175,6 +217,13 @@ public Long cachedIndexCount(final CountRequest countRequest) { Logger.warn(this.getClass(), String.format("OS Query: %s", countRequest)); Logger.warn(this.getClass(), String.format("Class %s: %s", e.getClass().getName(), exceptionMsg)); Logger.warn(this.getClass(), "----------------------------------------------"); + if (mustRaiseForPhase2Fallback()) { + // Not cached, for the same reason as the search path above. + // Index name only -- see the search path above for why the request is omitted. + throw new DotRuntimeException(String.format( + "OpenSearch count failed on index [%s]: %s", + countRequest.index(), exceptionMsg), e); + } if(shouldQueryCache(exceptionMsg)) { queryCache.put(countRequest, -1L); } @@ -204,10 +253,19 @@ public SearchHits searchHits(String query, int limit, int offset, String sortBy) try { indexToHit = inferIndexToHit(query); if (indexToHit == null) { + if (mustRaiseForPhase2Fallback()) { + // The query itself is omitted -- see cachedIndexSearch below. + throw new DotRuntimeException( + "Unable to determine which OpenSearch index to query"); + } return SearchHits.empty(); } - } catch (Exception e) { + } catch (final Exception e) { Logger.error(this, "Can't get indices information.", e); + if (mustRaiseForPhase2Fallback()) { + throw new DotRuntimeException( + "Can't get OpenSearch indices information: " + e.getMessage(), e); + } return SearchHits.empty(); } @@ -469,6 +527,11 @@ public PaginatedArrayList indexSearchScroll(String query, Stri Logger.warn(this.getClass(), String.format("OpenSearch error for query: %s", query)); Logger.warn(this.getClass(), String.format("Class %s: %s", e.getClass().getName(), exceptionMsg)); Logger.warn(this.getClass(), "----------------------------------------------"); + if (mustRaiseForPhase2Fallback()) { + // The Lucene query is omitted here for the same reason. + throw new DotRuntimeException( + "OpenSearch scroll failed: " + exceptionMsg, e); + } return new PaginatedArrayList<>(); } catch (final IllegalStateException e) { Logger.warnAndDebug(ContentFactoryIndexOperationsOS.class, e); diff --git a/dotCMS/src/test/java/com/dotcms/content/index/ContentFactoryIndexOperationsPhaseRoutingTest.java b/dotCMS/src/test/java/com/dotcms/content/index/ContentFactoryIndexOperationsPhaseRoutingTest.java new file mode 100644 index 000000000000..7d9563d6b902 --- /dev/null +++ b/dotCMS/src/test/java/com/dotcms/content/index/ContentFactoryIndexOperationsPhaseRoutingTest.java @@ -0,0 +1,475 @@ +package com.dotcms.content.index; + +import static com.dotcms.content.index.IndexConfigHelper.MigrationPhase.FLAG_KEY; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.assertSame; +import static org.junit.Assert.assertThrows; +import static org.mockito.ArgumentMatchers.anyBoolean; +import static org.mockito.ArgumentMatchers.anyInt; +import static org.mockito.ArgumentMatchers.anyString; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.times; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.verifyNoInteractions; +import static org.mockito.Mockito.when; + +import com.dotcms.content.index.IndexConfigHelper.MigrationPhase; +import com.dotcms.content.index.domain.SearchHits; +import com.dotcms.content.index.domain.TotalHits; +import com.dotmarketing.common.model.ContentletSearch; +import com.dotmarketing.exception.DotRuntimeException; +import com.dotmarketing.util.Config; +import com.dotmarketing.util.PaginatedArrayList; +import java.util.ArrayList; +import java.util.List; +import org.apache.logging.log4j.Level; +import org.apache.logging.log4j.LogManager; +import org.apache.logging.log4j.core.LogEvent; +import org.apache.logging.log4j.core.appender.AbstractAppender; +import org.junit.After; +import org.junit.Test; + +/** + * Unit tests for the per-phase read contract of the five {@link ContentFactoryIndexOperations} + * read operations that {@code ESContentFactoryImpl} funnels essentially all content search + * through (issue #37413). + * + *

What regressed

+ *

{@code ESContentFactoryImpl} picked a provider with a bare ternary and called it directly, + * so {@link PhaseRouter#read} — which holds the Phase 2 Elasticsearch fallback — was unreachable + * from the busiest read path in the product. With OpenSearch down in Phase 2 and Elasticsearch + * healthy and dual-written, {@code POST /api/content/_search} returned either a well-formed + * {@code 200} with zero results (indistinguishable from "this content type has no content") or + * a {@code 500}, instead of the result set Elasticsearch was holding the whole time.

+ * + *

Contract under test

+ *
+ * Phase │ Provider read │ On OpenSearch failure
+ * ──────┼───────────────┼──────────────────────────────────────────────────────
+ *   0   │ ES            │ n/a — OS is never contacted
+ *   1   │ ES            │ n/a — OS is never contacted
+ *   2   │ OS            │ logged at ERROR, retried against ES, ES result returned
+ *   3   │ OS            │ propagates — ES is decommissioned, there is no fallback
+ * 
+ * + *

Why this tests the router and not {@code ESContentFactoryImpl}

+ *

{@code ESContentFactoryImpl} cannot be constructed outside a container: its read methods + * run through static {@code CacheLocator}/{@code APILocator} calls — {@code indexCount} and + * {@code indexSearch} both go through the static {@code translateQuery} — before they reach a + * provider. So the per-phase contract is pinned here, against the real provider interface, and + * the separate question of whether {@code ESContentFactoryImpl} actually consults the router is + * proved by {@code ESContentFactoryImplPhase2FallbackTest} in {@code dotcms-integration}. A unit + * test on the router cannot show which class calls it.

+ * + * @author Fabrizzio Araya + */ +public class ContentFactoryIndexOperationsPhaseRoutingTest { + + /** The failure an unreachable node produces once the provider has wrapped it. */ + private static DotRuntimeException unreachable() { + return new DotRuntimeException("An error occurred when executing the Lucene Query", + new java.net.ConnectException("Connection refused")); + } + + @After + public void clearPhase() { + Config.setProperty(FLAG_KEY, null); + } + + private static void setPhase(final MigrationPhase phase) { + Config.setProperty(FLAG_KEY, String.valueOf(phase.ordinal())); + } + + private static PhaseRouter router( + final ContentFactoryIndexOperations es, final ContentFactoryIndexOperations os) { + return new PhaseRouter<>(es, os); + } + + // ========================================================================= + // indexCount — the operation that produced the 500s, because ContentHelper + // runs the count before the search and an uncached count threw. + // ========================================================================= + + /** + * Given : Phase 2, OpenSearch unreachable, Elasticsearch healthy and holding 184 documents. + * When : indexCount runs through the router. + * Then : the Elasticsearch count is returned — not 0, and not an exception. + */ + @Test + public void indexCount_phase2_osUnreachable_fallsBackToEs() { + setPhase(MigrationPhase.PHASE_2_DUAL_WRITE_OS_READS); + final ContentFactoryIndexOperations es = mock(ContentFactoryIndexOperations.class); + final ContentFactoryIndexOperations os = mock(ContentFactoryIndexOperations.class); + when(os.indexCount(anyString())).thenThrow(unreachable()); + when(es.indexCount(anyString())).thenReturn(184L); + + final long count = router(es, os).read(impl -> impl.indexCount("+contentType:Profile")); + + assertEquals("Phase 2 must serve the count Elasticsearch holds, not 0", 184L, count); + verify(os, times(1)).indexCount(anyString()); + verify(es, times(1)).indexCount(anyString()); + } + + /** + * Given : Phase 3, OpenSearch unreachable. + * When : indexCount runs through the router. + * Then : the failure propagates and Elasticsearch is never contacted — it is decommissioned + * in Phase 3, so a silent fallback there would report data that is genuinely gone. + */ + @Test + public void indexCount_phase3_osUnreachable_propagatesAndNeverTouchesEs() { + setPhase(MigrationPhase.PHASE_3_OPENSEARCH_ONLY); + final ContentFactoryIndexOperations es = mock(ContentFactoryIndexOperations.class); + final ContentFactoryIndexOperations os = mock(ContentFactoryIndexOperations.class); + when(os.indexCount(anyString())).thenThrow(unreachable()); + + assertThrows(DotRuntimeException.class, + () -> router(es, os).read(impl -> impl.indexCount("+contentType:Profile"))); + + verifyNoInteractions(es); + } + + /** + * Given : Phase 0 and Phase 1, where Elasticsearch serves reads. + * When : indexCount runs through the router. + * Then : Elasticsearch answers and OpenSearch is never contacted — the routing change must + * be a pure pass-through outside Phase 2. + */ + @Test + public void indexCount_phases0And1_readEsAndNeverTouchOs() { + for (final MigrationPhase phase : List.of(MigrationPhase.PHASE_0_MIGRATION_NOT_STARTED, + MigrationPhase.PHASE_1_DUAL_WRITE_ES_READS)) { + setPhase(phase); + final ContentFactoryIndexOperations es = mock(ContentFactoryIndexOperations.class); + final ContentFactoryIndexOperations os = mock(ContentFactoryIndexOperations.class); + when(es.indexCount(anyString())).thenReturn(27L); + + final long count = router(es, os).read(impl -> impl.indexCount("+contentType:JobPosting")); + + assertEquals("In " + phase + " the count must come from Elasticsearch", 27L, count); + verifyNoInteractions(os); + } + } + + // ========================================================================= + // searchHits — the operation behind indexSearch, and the one that produced + // the silent 200/total=0 once the legacy layer swallowed the failure. + // ========================================================================= + + /** + * Given : Phase 2, OpenSearch unreachable, Elasticsearch holding real hits. + * When : searchHits runs through the router. + * Then : the Elasticsearch hits are returned — the same object, so nothing substitutes an + * empty result on the way back. + */ + @Test + public void searchHits_phase2_osUnreachable_fallsBackToEs() { + setPhase(MigrationPhase.PHASE_2_DUAL_WRITE_OS_READS); + final ContentFactoryIndexOperations es = mock(ContentFactoryIndexOperations.class); + final ContentFactoryIndexOperations os = mock(ContentFactoryIndexOperations.class); + final SearchHits esHits = SearchHits.builder().totalHits(TotalHits.builder().value(184L).build()).build(); + when(os.searchHits(anyString(), anyInt(), anyInt(), anyString())).thenThrow(unreachable()); + when(es.searchHits(anyString(), anyInt(), anyInt(), anyString())).thenReturn(esHits); + + final SearchHits hits = router(es, os) + .read(impl -> impl.searchHits("+contentType:Profile +live:true", 7, 1, "title asc")); + + assertSame("Phase 2 must return the Elasticsearch hits untouched", esHits, hits); + verify(es, times(1)).searchHits(anyString(), anyInt(), anyInt(), anyString()); + } + + /** + * Given : Phase 3, OpenSearch unreachable. + * When : searchHits runs through the router. + * Then : the failure propagates; Elasticsearch is never contacted. + */ + @Test + public void searchHits_phase3_osUnreachable_propagates() { + setPhase(MigrationPhase.PHASE_3_OPENSEARCH_ONLY); + final ContentFactoryIndexOperations es = mock(ContentFactoryIndexOperations.class); + final ContentFactoryIndexOperations os = mock(ContentFactoryIndexOperations.class); + when(os.searchHits(anyString(), anyInt(), anyInt(), anyString())).thenThrow(unreachable()); + + assertThrows(DotRuntimeException.class, () -> router(es, os) + .read(impl -> impl.searchHits("+contentType:Hero", 10, 0, "title asc"))); + + verifyNoInteractions(es); + } + + // ========================================================================= + // search — inode-only search, behind findContentletsByHost + // ========================================================================= + + /** + * Given : Phase 2, OpenSearch unreachable, Elasticsearch holding two inodes. + * When : search runs through the router. + * Then : the Elasticsearch inodes are returned rather than an empty list. + */ + @Test + public void search_phase2_osUnreachable_fallsBackToEs() { + setPhase(MigrationPhase.PHASE_2_DUAL_WRITE_OS_READS); + final ContentFactoryIndexOperations es = mock(ContentFactoryIndexOperations.class); + final ContentFactoryIndexOperations os = mock(ContentFactoryIndexOperations.class); + when(os.search(anyString(), anyInt(), anyInt())).thenThrow(unreachable()); + when(es.search(anyString(), anyInt(), anyInt())).thenReturn(List.of("inode-a", "inode-b")); + + final List inodes = router(es, os) + .read(impl -> impl.search("+conhost:48190c8c", 10, 0)); + + assertEquals(List.of("inode-a", "inode-b"), inodes); + } + + // ========================================================================= + // indexSearchScroll — drains the whole scroll internally, so a failure means + // the operation produced nothing and can simply be re-run against ES. + // ========================================================================= + + /** + * Given : Phase 2, OpenSearch unreachable, Elasticsearch able to serve the scroll. + * When : indexSearchScroll runs through the router. + * Then : the Elasticsearch result is returned. This site can fall back safely precisely + * because it materialises the entire scroll before returning — there is no cursor + * left half-drained on the failing engine. + */ + @Test + public void indexSearchScroll_phase2_osUnreachable_fallsBackToEs() { + setPhase(MigrationPhase.PHASE_2_DUAL_WRITE_OS_READS); + final ContentFactoryIndexOperations es = mock(ContentFactoryIndexOperations.class); + final ContentFactoryIndexOperations os = mock(ContentFactoryIndexOperations.class); + final PaginatedArrayList esResults = new PaginatedArrayList<>(); + esResults.setTotalResults(317L); + when(os.indexSearchScroll(anyString(), anyString(), anyInt())).thenThrow(unreachable()); + when(es.indexSearchScroll(anyString(), anyString(), anyInt())).thenReturn(esResults); + + final PaginatedArrayList results = router(es, os) + .read(impl -> impl.indexSearchScroll("+contentType:Testimonials", "title asc", 1000)); + + assertEquals(317L, results.getTotalResults()); + verify(es, times(1)).indexSearchScroll(anyString(), anyString(), anyInt()); + } + + /** + * Given : Phase 3, OpenSearch unreachable. + * When : indexSearchScroll runs through the router. + * Then : the failure propagates. + */ + @Test + public void indexSearchScroll_phase3_osUnreachable_propagates() { + setPhase(MigrationPhase.PHASE_3_OPENSEARCH_ONLY); + final ContentFactoryIndexOperations es = mock(ContentFactoryIndexOperations.class); + final ContentFactoryIndexOperations os = mock(ContentFactoryIndexOperations.class); + when(os.indexSearchScroll(anyString(), anyString(), anyInt())).thenThrow(unreachable()); + + assertThrows(DotRuntimeException.class, () -> router(es, os) + .read(impl -> impl.indexSearchScroll("+contentType:Testimonials", "title asc", 1000))); + + verifyNoInteractions(es); + } + + // ========================================================================= + // createScrollQuery — routed for provider selection only. + // ========================================================================= + + /** + * Given : Phase 2 and a working OpenSearch provider. + * When : createScrollQuery runs through the router. + * Then : the OpenSearch cursor is returned and Elasticsearch is never contacted. + * + *

This site is routed so that provider selection has exactly one mechanism in the class, + * but its fallback can never fire and that is correct, not an oversight: the OpenSearch + * implementation is {@code return new OSContentletScrollImpl(...)} — pure construction, no + * I/O, nothing to throw. The requests happen later, inside the returned cursor, outside the + * router. A mid-scroll fallback is impossible in principle anyway, because an OpenSearch + * scroll id is meaningless to Elasticsearch, so a half-drained scroll cannot be resumed on + * the other engine. The residual gap is documented at the call site.

+ */ + @Test + public void createScrollQuery_phase2_selectsOsAndDoesNotTouchEs() { + setPhase(MigrationPhase.PHASE_2_DUAL_WRITE_OS_READS); + final ContentFactoryIndexOperations es = mock(ContentFactoryIndexOperations.class); + final ContentFactoryIndexOperations os = mock(ContentFactoryIndexOperations.class); + final IndexContentletScroll osScroll = mock(IndexContentletScroll.class); + when(os.createScrollQuery(anyString(), any(), anyBoolean(), anyInt(), anyString())) + .thenReturn(osScroll); + + final IndexContentletScroll scroll = router(es, os).read(impl -> + impl.createScrollQuery("+contentType:Profile", null, false, 100, "title asc")); + + assertSame(osScroll, scroll); + verifyNoInteractions(es); + } + + /** + * Given : Phase 1, where Elasticsearch serves reads. + * When : createScrollQuery runs through the router. + * Then : the Elasticsearch cursor is returned and OpenSearch is never contacted. + */ + @Test + public void createScrollQuery_phase1_selectsEs() { + setPhase(MigrationPhase.PHASE_1_DUAL_WRITE_ES_READS); + final ContentFactoryIndexOperations es = mock(ContentFactoryIndexOperations.class); + final ContentFactoryIndexOperations os = mock(ContentFactoryIndexOperations.class); + final IndexContentletScroll esScroll = mock(IndexContentletScroll.class); + when(es.createScrollQuery(anyString(), any(), anyBoolean(), anyInt(), anyString())) + .thenReturn(esScroll); + + final IndexContentletScroll scroll = router(es, os).read(impl -> + impl.createScrollQuery("+contentType:Profile", null, false, 100, "title asc")); + + assertSame(esScroll, scroll); + verifyNoInteractions(os); + } + + // ========================================================================= + // The fallback must be a single attempt — a failing read costs one extra + // call against ES, with no nesting and no retry storm. + // ========================================================================= + + /** + * Given : Phase 2 with both engines failing on the same read. + * When : the read runs through the router. + * Then : the Elasticsearch failure surfaces to the caller, and each provider was called + * exactly once. A two-engine outage is the one case where a caller now sees an error + * where it previously saw an empty result — which is the point of the issue, since + * the silent-empty variant is the dangerous one. + */ + @Test + public void phase2_bothEnginesFail_surfacesErrorAfterExactlyOneAttemptEach() { + setPhase(MigrationPhase.PHASE_2_DUAL_WRITE_OS_READS); + final ContentFactoryIndexOperations es = mock(ContentFactoryIndexOperations.class); + final ContentFactoryIndexOperations os = mock(ContentFactoryIndexOperations.class); + when(os.indexCount(anyString())).thenThrow(unreachable()); + when(es.indexCount(anyString())).thenThrow(unreachable()); + + assertThrows(DotRuntimeException.class, + () -> router(es, os).read(impl -> impl.indexCount("+contentType:Profile"))); + + verify(os, times(1)).indexCount(anyString()); + verify(es, times(1)).indexCount(anyString()); + } + + /** + * Given : Phase 2 and a healthy OpenSearch. + * When : the read succeeds. + * Then : Elasticsearch is never contacted. The fallback must cost nothing on the success + * path — otherwise Phase 2 would double every read in the product. + */ + @Test + public void phase2_osHealthy_neverTouchesEs() { + setPhase(MigrationPhase.PHASE_2_DUAL_WRITE_OS_READS); + final ContentFactoryIndexOperations es = mock(ContentFactoryIndexOperations.class); + final ContentFactoryIndexOperations os = mock(ContentFactoryIndexOperations.class); + when(os.indexCount(anyString())).thenReturn(215L); + + final long count = router(es, os).read(impl -> impl.indexCount("+contentType:Hero")); + + assertEquals(215L, count); + + verify(es, never()).indexCount(anyString()); + } + + // ========================================================================= + // The fallback log line (AC-003). + // + // The pre-fix path logged these failures at WARN, which is why an outage was + // invisible to log-based monitoring even though the design documents it as the + // early-warning signal. Level is therefore part of the contract, not cosmetic: + // a test that only checked "something was logged" would pass against the old + // behaviour. + // ========================================================================= + + /** Captures events published to a single log4j2 logger. */ + private static final class CapturingAppender extends AbstractAppender { + + private final List events = new ArrayList<>(); + + CapturingAppender() { + super("capture-phase-router", null, null, true, null); + } + + @Override + public void append(final LogEvent event) { + events.add(event.toImmutable()); + } + } + + private CapturingAppender attachAppender() { + final CapturingAppender appender = new CapturingAppender(); + appender.start(); + final org.apache.logging.log4j.core.Logger logger = + (org.apache.logging.log4j.core.Logger) LogManager.getLogger(PhaseRouter.class); + logger.addAppender(appender); + return appender; + } + + private void detachAppender(final CapturingAppender appender) { + final org.apache.logging.log4j.core.Logger logger = + (org.apache.logging.log4j.core.Logger) LogManager.getLogger(PhaseRouter.class); + logger.removeAppender(appender); + appender.stop(); + } + + /** + * Given : Phase 2 and an unreachable OpenSearch. + * When : a named read falls back. + * Then : exactly one event is logged, at ERROR, naming both the operation that failed and + * the cause, with the throwable attached. One event per read — not per provider, and + * with no retry storm behind it. + */ + @Test + public void phase2_fallback_logsOnceAtErrorNamingOperationAndCause() { + setPhase(MigrationPhase.PHASE_2_DUAL_WRITE_OS_READS); + final ContentFactoryIndexOperations es = mock(ContentFactoryIndexOperations.class); + final ContentFactoryIndexOperations os = mock(ContentFactoryIndexOperations.class); + when(os.indexCount(anyString())).thenThrow(unreachable()); + when(es.indexCount(anyString())).thenReturn(184L); + + final CapturingAppender appender = attachAppender(); + try { + router(es, os).read("indexCount", impl -> impl.indexCount("+contentType:Profile")); + } finally { + detachAppender(appender); + } + + assertEquals("A fallback must log exactly one event per read", 1, appender.events.size()); + final LogEvent event = appender.events.get(0); + assertEquals("The fallback must be ERROR — WARN is what monitoring was missing", + Level.ERROR, event.getLevel()); + final String message = event.getMessage().getFormattedMessage(); + assertTrue("The log line must name the failing operation, so monitoring can say what " + + "stopped working. Was: " + message, + message.contains("indexCount")); + assertTrue("The log line must carry the cause. Was: " + message, + message.contains("Connection refused")); + assertNotNull("The throwable must be attached so the stack identifies the call site", + event.getThrown()); + } + + /** + * Given : Phase 3 and an unreachable OpenSearch. + * When : the read propagates instead of falling back. + * Then : no fallback event is logged. A fallback log line in Phase 3 would be a false + * signal — there is no fallback there, and Elasticsearch is decommissioned. + */ + @Test + public void phase3_failure_logsNoFallbackEvent() { + setPhase(MigrationPhase.PHASE_3_OPENSEARCH_ONLY); + final ContentFactoryIndexOperations es = mock(ContentFactoryIndexOperations.class); + final ContentFactoryIndexOperations os = mock(ContentFactoryIndexOperations.class); + when(os.indexCount(anyString())).thenThrow(unreachable()); + + final CapturingAppender appender = attachAppender(); + try { + assertThrows(DotRuntimeException.class, () -> router(es, os) + .read("indexCount", impl -> impl.indexCount("+contentType:Profile"))); + } finally { + detachAppender(appender); + } + + assertEquals("Phase 3 must not report a fallback", 0, appender.events.size()); + } +} diff --git a/dotcms-integration/src/test/java/com/dotcms/MainSuite1b.java b/dotcms-integration/src/test/java/com/dotcms/MainSuite1b.java index f9332c46ce24..4f1369b8535b 100644 --- a/dotcms-integration/src/test/java/com/dotcms/MainSuite1b.java +++ b/dotcms-integration/src/test/java/com/dotcms/MainSuite1b.java @@ -96,6 +96,8 @@ com.dotcms.content.elasticsearch.business.ESIndexSpeedTest.class, com.dotcms.content.elasticsearch.business.ES6UpgradeTest.class, com.dotcms.content.elasticsearch.business.ESContentFactoryImplTest.class, + com.dotcms.content.elasticsearch.business.ESContentFactoryImplPhase2FallbackTest.class, + com.dotcms.content.elasticsearch.business.ESContentFactoryImplMissingOsIndexTest.class, com.dotcms.graphql.datafetcher.page.ContentMapDataFetcherTest.class, com.dotcms.graphql.datafetcher.RelationshipFieldDataFetcherTest.class, com.dotcms.rest.StoryBlockMarkdownPopulatorTest.class, diff --git a/dotcms-integration/src/test/java/com/dotcms/content/elasticsearch/business/ESContentFactoryImplMissingOsIndexTest.java b/dotcms-integration/src/test/java/com/dotcms/content/elasticsearch/business/ESContentFactoryImplMissingOsIndexTest.java new file mode 100644 index 000000000000..638a2c343e13 --- /dev/null +++ b/dotcms-integration/src/test/java/com/dotcms/content/elasticsearch/business/ESContentFactoryImplMissingOsIndexTest.java @@ -0,0 +1,299 @@ +package com.dotcms.content.elasticsearch.business; + +import static com.dotcms.content.index.IndexConfigHelper.MigrationPhase.FLAG_KEY; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNotEquals; +import static org.junit.Assert.assertTrue; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +import com.dotcms.content.index.IndexConfigHelper.MigrationPhase; +import com.dotcms.content.index.domain.SearchHits; +import com.dotcms.content.index.opensearch.ContentFactoryIndexOperationsOS; +import com.dotcms.content.index.opensearch.OSClientProvider; +import com.dotcms.content.index.opensearch.OSQueryCache; +import com.dotcms.contenttype.model.type.ContentType; +import com.dotcms.datagen.ContentTypeDataGen; +import com.dotcms.datagen.ContentletDataGen; +import com.dotcms.util.IntegrationTestInitService; +import com.dotmarketing.business.CacheLocator; +import com.dotmarketing.portlets.contentlet.model.Contentlet; +import com.dotmarketing.portlets.contentlet.model.IndexPolicy; +import com.dotmarketing.util.Config; +import java.util.Optional; +import org.junit.After; +import org.junit.BeforeClass; +import org.junit.Test; +import org.opensearch.client.opensearch.OpenSearchClient; +import org.opensearch.client.opensearch._types.ErrorCause; +import org.opensearch.client.opensearch._types.ErrorResponse; +import org.opensearch.client.opensearch._types.OpenSearchException; +import org.opensearch.client.opensearch.core.CountRequest; +import org.opensearch.client.opensearch.core.SearchRequest; + +/** + * Integration tests for the second failure class behind + * #37413: an OpenSearch node that is + * reachable but answers with an error — chiefly a missing counterpart index (AC-005), + * which also covers the reactivated pre-migration backup index scenario. + * + *

Why this is a separate failure class from the routing fix

+ *

Routing the read path through the phase router fixes an unreachable OpenSearch: a + * connection failure reaches the provider's generic handler and is rethrown as a runtime + * exception the router catches. A reachable node answering {@code index_not_found_exception} is + * different. That is an {@code OpenSearchException}, which the provider absorbs into a sentinel — + * an empty {@code ERROR_HIT} result for a search, {@code -1} for a count. The provider then + * reports success, the router sees nothing to catch, and the caller receives a legitimate-looking + * empty result. Routing alone does not fix it.

+ * + *

Why the OpenSearch client is mocked here

+ *

An earlier version of this test relied on the container simply having no OpenSearch + * counterpart indices. That looked like the real customer scenario but was not: the failure it + * produced was dotCMS's own index-name resolution giving up ("Unable to load default versioned + * indices"), which is a {@code DotRuntimeException} thrown before the OpenSearch client + * is ever called — so it exercised the routing fix again rather than the sentinel branch, and + * {@code index_not_found_exception} never appeared once in the run. Mocking the client is what + * makes the failure the one AC-005 is actually about.

+ * + *

Coverage this test cannot give

+ *

The scroll path carries a second, independent swallow of the same kind — its own + * handler that logs a warning and returns an empty list. It cannot be driven from here: the + * scroll resolves its client through CDI rather than through the injected provider, so a mocked + * client does not reach it. The fix still covers it; only the behavioural proof is missing, and + * that gap is deliberate and recorded rather than silently skipped.

+ * + * @author Fabrizzio Araya + */ +public class ESContentFactoryImplMissingOsIndexTest { + + private static String query; + + @BeforeClass + public static void prepare() throws Exception { + IntegrationTestInitService.getInstance().init(); + final ContentType contentType = new ContentTypeDataGen().nextPersisted(); + final Contentlet contentlet = new ContentletDataGen(contentType.id()) + .setPolicy(IndexPolicy.WAIT_FOR).nextPersistedAndPublish(); + assertTrue("Test setup must produce a persisted contentlet", + null != contentlet.getInode()); + query = "+contentType:" + contentType.variable() + " +live:true"; + } + + @After + public void clearPhase() { + Config.setProperty(FLAG_KEY, null); + } + + private static void setPhase(final MigrationPhase phase) { + Config.setProperty(FLAG_KEY, String.valueOf(phase.ordinal())); + } + + /** The 404 a reachable OpenSearch returns when the counterpart index does not exist. */ + private static OpenSearchException indexNotFound() { + return new OpenSearchException(ErrorResponse.of(r -> r + .status(404) + .error(ErrorCause.of(c -> c + .type("index_not_found_exception") + .reason("no such index [working_20260101000000.os]"))))); + } + + /** + * An OpenSearch provider whose node is up and answering, but whose counterpart index is gone. + * The query cache is mocked empty so every call reaches the client rather than a cached hit — + * a cached result would mask the failure exactly the way it masked it in QA. + */ + private static ContentFactoryIndexOperationsOS openSearchMissingIndex() throws Exception { + return openSearchMissingIndex(mock(OSQueryCache.class)); + } + + /** Same, with a caller-supplied cache mock so a test can assert what was written to it. */ + private static ContentFactoryIndexOperationsOS openSearchMissingIndex( + final OSQueryCache queryCache) throws Exception { + final OpenSearchClient client = mock(OpenSearchClient.class); + when(client.search(any(SearchRequest.class), any())).thenThrow(indexNotFound()); + when(client.count(any(CountRequest.class))).thenThrow(indexNotFound()); + + final OSClientProvider clientProvider = mock(OSClientProvider.class); + when(clientProvider.getClient()).thenReturn(client); + + when(queryCache.get(any(SearchRequest.class))).thenReturn(Optional.empty()); + when(queryCache.get(any(CountRequest.class))).thenReturn(Optional.empty()); + + return new ContentFactoryIndexOperationsOS(queryCache, clientProvider); + } + + /** The factory with a real Elasticsearch leg and an OpenSearch leg missing its index. */ + private static ESContentFactoryImpl factory() throws Exception { + return new ESContentFactoryImpl( + new ContentFactoryIndexOperationsES(CacheLocator.getESQueryCache()), + openSearchMissingIndex()); + } + + /** + * Method to test: {@link ContentFactoryIndexOperationsOS#indexCount(String)} + * Given Scenario: the OpenSearch provider is asked directly, with its index missing. + * Expected Result: it does NOT return the right count — it either throws or hands back its + * sentinel. + * + *

This is the precondition the rest of the class depends on, and it asserts the failure + * is real rather than assuming it. Without it a green run could mean "the fallback works" or + * "nothing ever failed", and those must not be indistinguishable — the previous version of + * this class passed for exactly that reason while testing the wrong branch.

+ */ + @Test + public void precondition_openSearchCannotServeTheQuery() throws Exception { + long osCount = Long.MIN_VALUE; + try { + osCount = openSearchMissingIndex().indexCount(query); + } catch (final Exception e) { + return; // throwing is one of the two acceptable outcomes + } + assertNotEquals("OpenSearch answered the query correctly, so the missing-index scenario " + + "is not being exercised and the rest of this class would pass vacuously", + 1L, osCount); + } + + /** + * Method to test: {@link ESContentFactoryImpl#indexSearch(String, int, int, String)} + * Given Scenario: Phase 2, OpenSearch reachable but answering + * {@code index_not_found_exception}, Elasticsearch healthy and holding the + * content. + * Expected Result: the real Elasticsearch hits are served. Zero hits here is the silent + * variant of the outage — the dangerous one, because it is indistinguishable + * from an empty content type. + */ + @Test + public void indexSearch_phase2_missingOsIndex_fallsBackToEs() throws Exception { + setPhase(MigrationPhase.PHASE_2_DUAL_WRITE_OS_READS); + + final SearchHits hits = factory().indexSearch(query, 10, 0, "modDate desc"); + + assertEquals("Phase 2 must serve the hits Elasticsearch holds when the OpenSearch " + + "counterpart index is missing", 1L, hits.getTotalHits().value()); + } + + /** + * Method to test: {@link ESContentFactoryImpl#indexCount(String)} + * Given Scenario: Phase 2, OpenSearch answering {@code index_not_found_exception}. + * Expected Result: the Elasticsearch count is served. A count of 0 is the silent variant; + * -1 is the sentinel leaking to the caller. + */ + @Test + public void indexCount_phase2_missingOsIndex_fallsBackToEs() throws Exception { + setPhase(MigrationPhase.PHASE_2_DUAL_WRITE_OS_READS); + + final long count = factory().indexCount(query); + + assertEquals("Phase 2 must serve the count Elasticsearch holds when the OpenSearch " + + "counterpart index is missing", 1L, count); + } + + /** + * Method to test: {@link ESContentFactoryImpl#indexCount(String)} + * Given Scenario: Phase 3, OpenSearch answering {@code index_not_found_exception}. + * Elasticsearch is decommissioned in Phase 3. + * Expected Result: the caller must NOT receive Elasticsearch data. Either the sentinel or an + * exception is correct; silently serving Elasticsearch results would be a + * worse bug than the one being fixed, because it would report data absent + * from the only live engine. + */ + @Test + public void indexCount_phase3_missingOsIndex_neverFallsBackToEs() throws Exception { + setPhase(MigrationPhase.PHASE_3_OPENSEARCH_ONLY); + + long count = Long.MIN_VALUE; + try { + count = factory().indexCount(query); + } catch (final Exception e) { + return; // propagating is a correct Phase 3 outcome + } + assertNotEquals("Phase 3 must not fall back to Elasticsearch — it is decommissioned " + + "there, so serving its data would hide that OpenSearch has no index", + 1L, count); + } + + /** + * Method to test: {@link ESContentFactoryImpl#indexSearch(String, int, int, String)} + * Given Scenario: Phase 3, OpenSearch answering {@code index_not_found_exception}. + * Expected Result: no Elasticsearch data, by the same reasoning as the count above. + */ + @Test + public void indexSearch_phase3_missingOsIndex_neverFallsBackToEs() throws Exception { + setPhase(MigrationPhase.PHASE_3_OPENSEARCH_ONLY); + + SearchHits hits = null; + try { + hits = factory().indexSearch(query, 10, 0, "modDate desc"); + } catch (final Exception e) { + return; // propagating is a correct Phase 3 outcome + } + assertNotEquals("Phase 3 must not fall back to Elasticsearch", + 1L, hits.getTotalHits().value()); + } + + /** + * Method to test: {@link ESContentFactoryImpl#indexCount(String)} + * Given Scenario: Phase 1, where Elasticsearch serves reads and OpenSearch is a shadow. + * Expected Result: the count is correct and the missing OpenSearch index is irrelevant. The + * change must be a pure pass-through outside Phase 2 — these call sites + * carry essentially all content search, so any behaviour change here is + * felt site-wide in every phase. + */ + @Test + public void indexCount_phase1_missingOsIndexIsIrrelevant() throws Exception { + setPhase(MigrationPhase.PHASE_1_DUAL_WRITE_ES_READS); + + assertEquals("Phase 1 reads Elasticsearch regardless of OpenSearch index state", + 1L, factory().indexCount(query)); + } + + /** + * Method to test: {@link ESContentFactoryImpl#indexCount(String)} + * Given Scenario: Phase 0, the default. + * Expected Result: unchanged. Elasticsearch answers; the OpenSearch leg is never consulted. + */ + @Test + public void indexCount_phase0_missingOsIndexIsIrrelevant() throws Exception { + setPhase(MigrationPhase.PHASE_0_MIGRATION_NOT_STARTED); + + assertEquals("Phase 0 behaviour must be untouched", + 1L, factory().indexCount(query)); + } + + /** + * Method to test: {@link ESContentFactoryImpl#indexSearch(String, int, int, String)} + * Given Scenario: Phase 2, OpenSearch answering with an error, and the fallback firing. + * Expected Result: nothing is written to the OpenSearch query cache. + * + *

This is the half of the query-cache concern that is a real defect rather than a + * reproduction hazard. The provider caches its error sentinel for some failure messages; on + * the Phase 2 path such an entry would be replayed to every later identical query as a + * successful empty result, outliving the outage and defeating the fallback even after + * OpenSearch recovers. The raise therefore has to happen before the cache write, and this + * asserts the order rather than trusting it.

+ * + *

A cache hit during an outage is not a defect and is deliberately not tested + * here: the cached value is a real result from a real earlier query, so serving it is + * correct. It matters only because it hides the failure from a human running the + * reproduction — which cuts both ways, since it can just as easily produce a false + * confirmation that a fix works.

+ */ + @Test + public void phase2_fallback_neverWritesTheErrorSentinelToTheQueryCache() throws Exception { + setPhase(MigrationPhase.PHASE_2_DUAL_WRITE_OS_READS); + final OSQueryCache queryCache = mock(OSQueryCache.class); + + final SearchHits hits = new ESContentFactoryImpl( + new ContentFactoryIndexOperationsES(CacheLocator.getESQueryCache()), + openSearchMissingIndex(queryCache)) + .indexSearch(query, 10, 0, "modDate desc"); + + assertEquals("Precondition: the fallback must have served the Elasticsearch hits", + 1L, hits.getTotalHits().value()); + verify(queryCache, never()).put(any(SearchRequest.class), any()); + verify(queryCache, never()).put(any(CountRequest.class), any()); + } +} diff --git a/dotcms-integration/src/test/java/com/dotcms/content/elasticsearch/business/ESContentFactoryImplPhase2FallbackTest.java b/dotcms-integration/src/test/java/com/dotcms/content/elasticsearch/business/ESContentFactoryImplPhase2FallbackTest.java new file mode 100644 index 000000000000..b353dcccb92f --- /dev/null +++ b/dotcms-integration/src/test/java/com/dotcms/content/elasticsearch/business/ESContentFactoryImplPhase2FallbackTest.java @@ -0,0 +1,218 @@ +package com.dotcms.content.elasticsearch.business; + +import static com.dotcms.content.index.IndexConfigHelper.MigrationPhase.FLAG_KEY; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; + +import com.dotcms.content.index.ContentFactoryIndexOperations; +import com.dotcms.content.index.IndexConfigHelper.MigrationPhase; +import com.dotcms.content.index.IndexContentletScroll; +import com.dotcms.content.index.domain.SearchHits; +import com.dotcms.contenttype.model.type.ContentType; +import com.dotcms.datagen.ContentTypeDataGen; +import com.dotcms.datagen.ContentletDataGen; +import com.dotcms.util.IntegrationTestInitService; +import com.dotmarketing.business.CacheLocator; +import com.dotmarketing.common.model.ContentletSearch; +import com.dotmarketing.exception.DotRuntimeException; +import com.dotmarketing.portlets.contentlet.model.Contentlet; +import com.dotmarketing.portlets.contentlet.model.IndexPolicy; +import com.dotmarketing.util.Config; +import com.dotmarketing.util.PaginatedArrayList; +import com.liferay.portal.model.User; +import java.util.List; +import org.junit.After; +import org.junit.BeforeClass; +import org.junit.Test; + +/** + * Integration tests proving that {@link ESContentFactoryImpl} routes its reads through + * {@link com.dotcms.content.index.PhaseRouter}, so the Phase 2 Elasticsearch fallback actually + * fires for content search (#37413). + * + *

What regressed

+ *

The fallback was implemented and correct; the content read path simply never reached it. + * {@code ESContentFactoryImpl} selected a provider with a bare ternary and called it directly + * from five read call sites, so the router was unreachable from the busiest read path in the + * product. In Phase 2 with OpenSearch down and Elasticsearch healthy and dual-written, + * {@code POST /api/content/_search} returned a well-formed {@code 200} with zero results — which + * a caller cannot tell apart from "this content type has no content", so a live site rendered as + * missing content with nothing for 5xx monitoring to catch.

+ * + *

What this test adds over the unit tests

+ *

{@code ContentFactoryIndexOperationsPhaseRoutingTest} pins the per-phase fallback contract + * against the provider interface, but a unit test on the router cannot show which class + * calls it — and that was the entire defect. This test is the wiring proof: it injects an + * OpenSearch provider that fails every read alongside the real Elasticsearch provider, and + * asserts that the factory's own read methods still return real Elasticsearch data. Before the + * fix these calls propagate the injected failure, because nothing catches it.

+ * + * @author Fabrizzio Araya + */ +public class ESContentFactoryImplPhase2FallbackTest { + + private static ContentType contentType; + private static String query; + + @BeforeClass + public static void prepare() throws Exception { + IntegrationTestInitService.getInstance().init(); + contentType = new ContentTypeDataGen().nextPersisted(); + // WAIT_FOR so the document is searchable before the assertions run. + final Contentlet contentlet = new ContentletDataGen(contentType.id()) + .setPolicy(IndexPolicy.WAIT_FOR).nextPersistedAndPublish(); + assertTrue("Test setup must produce a persisted contentlet", + null != contentlet.getInode()); + query = "+contentType:" + contentType.variable() + " +live:true"; + } + + @After + public void clearPhase() { + Config.setProperty(FLAG_KEY, null); + } + + private static void setPhase(final MigrationPhase phase) { + Config.setProperty(FLAG_KEY, String.valueOf(phase.ordinal())); + } + + /** + * An OpenSearch provider standing in for an unreachable node: every read fails the way the + * real provider's generic handler wraps a connection failure. + */ + private static final class UnreachableOpenSearch implements ContentFactoryIndexOperations { + + private static DotRuntimeException down() { + return new DotRuntimeException("An error occurred when executing the Lucene Query", + new java.net.ConnectException("Connection refused")); + } + + @Override + public String inferIndexToHit(final String query) { + throw down(); + } + + @Override + public long indexCount(final String query) { + throw down(); + } + + @Override + public SearchHits searchHits(final String query, final int limit, final int offset, + final String sortBy) { + throw down(); + } + + @Override + public List search(final String query, final int limit, final int offset) { + throw down(); + } + + @Override + public PaginatedArrayList indexSearchScroll(final String query, + final String sortBy, final int scrollBatchSize) { + throw down(); + } + + @Override + public IndexContentletScroll createScrollQuery(final String luceneQuery, final User user, + final boolean respectFrontendRoles, final int batchSize, final String sortBy) { + throw down(); + } + + @Override + public IndexContentletScroll createScrollQuery(final String luceneQuery, final User user, + final boolean respectFrontendRoles, final int batchSize) { + throw down(); + } + } + + /** A factory whose OpenSearch leg is dead and whose Elasticsearch leg is the real one. */ + private static ESContentFactoryImpl factoryWithDeadOpenSearch() { + return new ESContentFactoryImpl( + new ContentFactoryIndexOperationsES(CacheLocator.getESQueryCache()), + new UnreachableOpenSearch()); + } + + /** + * Method to test: {@link ESContentFactoryImpl#indexCount(String)} + * Given Scenario: Phase 2, OpenSearch unreachable, Elasticsearch healthy and holding the + * content. This is the call that produced the 500s, because the count runs + * before the search and an uncached count threw. + * Expected Result: the real Elasticsearch count comes back — not 0, and not an exception. + */ + @Test + public void indexCount_phase2_deadOpenSearch_servesElasticsearchCount() { + setPhase(MigrationPhase.PHASE_2_DUAL_WRITE_OS_READS); + + final long count = factoryWithDeadOpenSearch().indexCount(query); + + assertEquals("Phase 2 must fall back to the count Elasticsearch holds", 1L, count); + } + + /** + * Method to test: {@link ESContentFactoryImpl#indexSearch(String, int, int, String)} + * Given Scenario: Phase 2, OpenSearch unreachable, Elasticsearch healthy. + * Expected Result: the real hits come back. A zero-hit result here is the silent variant of + * the outage — the dangerous one, because it looks like an empty content + * type rather than a failure. + */ + @Test + public void indexSearch_phase2_deadOpenSearch_servesElasticsearchHits() { + setPhase(MigrationPhase.PHASE_2_DUAL_WRITE_OS_READS); + + final SearchHits hits = factoryWithDeadOpenSearch() + .indexSearch(query, 10, 0, "modDate desc"); + + assertEquals("Phase 2 must fall back to the hits Elasticsearch holds", + 1L, hits.getTotalHits().value()); + } + + /** + * Method to test: {@link ESContentFactoryImpl#indexSearchScroll(String, String)} + * Given Scenario: Phase 2, OpenSearch unreachable, Elasticsearch healthy. + * Expected Result: the scroll is served from Elasticsearch. This site can fall back safely + * because it materialises the whole scroll before returning, so a failure + * leaves no cursor half-drained on the failing engine. + */ + @Test + public void indexSearchScroll_phase2_deadOpenSearch_servesElasticsearchResults() { + setPhase(MigrationPhase.PHASE_2_DUAL_WRITE_OS_READS); + + final PaginatedArrayList results = + factoryWithDeadOpenSearch().indexSearchScroll(query, "modDate desc"); + + assertEquals("Phase 2 must fall back to the scroll Elasticsearch can serve", + 1, results.size()); + } + + /** + * Method to test: {@link ESContentFactoryImpl#indexCount(String)} + * Given Scenario: Phase 1, where Elasticsearch serves reads and OpenSearch is only a shadow. + * Expected Result: the count is served without the dead OpenSearch provider being consulted + * at all. The routing change has to be a pure pass-through outside Phase 2 — + * these five call sites carry essentially all content search in the product, + * so any behaviour change here is felt site-wide in every phase. + */ + @Test + public void indexCount_phase1_deadOpenSearchIsNeverConsulted() { + setPhase(MigrationPhase.PHASE_1_DUAL_WRITE_ES_READS); + + final long count = factoryWithDeadOpenSearch().indexCount(query); + + assertEquals("Phase 1 reads Elasticsearch and must not touch OpenSearch", 1L, count); + } + + /** + * Method to test: {@link ESContentFactoryImpl#indexCount(String)} + * Given Scenario: Phase 0, the default — migration not started. + * Expected Result: same as Phase 1. Elasticsearch answers; OpenSearch is never contacted. + */ + @Test + public void indexCount_phase0_deadOpenSearchIsNeverConsulted() { + setPhase(MigrationPhase.PHASE_0_MIGRATION_NOT_STARTED); + + final long count = factoryWithDeadOpenSearch().indexCount(query); + + assertEquals("Phase 0 reads Elasticsearch and must not touch OpenSearch", 1L, count); + } +} diff --git a/specs/37413-phase2-read-fallback/spec.md b/specs/37413-phase2-read-fallback/spec.md new file mode 100644 index 000000000000..f66e50c141e5 --- /dev/null +++ b/specs/37413-phase2-read-fallback/spec.md @@ -0,0 +1,249 @@ +# Issue Resolution Specification: Phase 2 ES read fallback never fires for content search + +**Feature Branch**: `37413-phase2-read-fallback` + +**Created**: 2026-09-07 + +**Status**: Draft + +**Type**: Issue / Bug Resolution + +**Related GitHub Issue**: #37413 + +**Input**: User description: "37413" + +## Problem Statement *(mandatory)* + +The ES→OpenSearch migration design promises an automatic **read fallback to Elasticsearch in +Phase 2**: OpenSearch serves reads, Elasticsearch is still live and in sync, and if OpenSearch +throws on a read the error is logged at `ERROR` and the read is retried against Elasticsearch. +That fallback is implemented, but the **content read path never reaches it**, so it never fires. + +With OpenSearch unavailable in Phase 2 and Elasticsearch up and holding a complete copy of the +data, `POST /api/content/_search` does not fall back. It returns one of two things: + +- **HTTP 200 with `resultsSize: 0`** — a well-formed empty result. A caller cannot tell this + apart from "this content type has no content", so a live site renders as **missing content** + with no error surfaced anywhere and nothing for 5xx monitoring to catch. +- **HTTP 500** — for content types whose count was not already cached. + +**Severity / Impact**: High. Every customer running Phase 2 loses the documented safety net — +a transient OpenSearch blip, a slow node, or a missing OpenSearch index takes down content +delivery instead of degrading to Elasticsearch. The silent-empty variant is the more dangerous +of the two. Beyond the outage case, this also blocks recommending Phase 2 to customers at all: +Cloud and Support are currently being told "Phase 2 is safe, OpenSearch failures fall back to +Elasticsearch", and for content search that is false today. + +**This is not a regression.** With its read engine down dotCMS has always behaved this way — +Phase 0 with Elasticsearch down does the same thing through the same legacy code. What is +broken is the *promise*, not previously-working behavior. For content search the fallback +wiring was never written. + +## Reproduction *(mandatory)* + +**Environment**: `main` @ `788795e915` (2026-09-03). Local instance in Phase 2 +(`DOT_FEATURE_FLAG_OPEN_SEARCH_PHASE=2`), Elasticsearch 7.10.2 and OpenSearch 3.4.0, real +content indexed and in sync on both engines. + +**Steps to Reproduce**: + +1. Confirm the live phase from `GET /api/v1/jvm` (`DOT_FEATURE_FLAG_OPEN_SEARCH_PHASE` = `2`) + and that `GET /api/v1/index/migration/readiness` reports `readEngine: OpenSearch`. +2. Stop the OpenSearch node (stop the container or block its port). Leave Elasticsearch running + and healthy. +3. Call `POST /api/content/_search` for several content types that **do** have live content: + `{"query":"+contentType:Profile +live:true","limit":7,"offset":1}` +4. **Vary `offset` on every call.** An identical request body is served from the query cache and + returns the pre-outage result, which looks exactly like a working fallback. Only novel, + uncached queries expose the defect — this is how it was nearly missed in QA. +5. Observe the responses, then restart OpenSearch and repeat to confirm the content was present + the whole time. + +**Expected Behavior**: Every query returns the result set Elasticsearch holds, and each fallback +is logged once at `ERROR` naming the failing OpenSearch operation and its cause. + +**Actual Behavior**: + +| Content type | Actual with OpenSearch down | Same call, OpenSearch up | +|---|---|---| +| Profile | **200, `total=0`** | 200, `total=184` | +| Testimonials | **200, `total=0`** | 200, `total=317` | +| JobPosting | **200, `total=0`** | 200, `total=27` | +| Hero | **500** | 200, `total=215` | +| ContactCard | **500** | 200, `total=106` | + +Failures are logged at **`WARN`**, not the `ERROR` the design specifies, so the "early-warning +signal per read" is weaker than documented. + +**Reproducibility**: Always, on demand — provided the query is not already in the query cache +(see step 4). + +## Scope of Investigation *(mandatory)* + +- **Affected area**: Content search / delivery read path (`/api/content/_search`, `ContentletAPI` + index reads, Velocity `$dotcontent.pull`), under the ES→OpenSearch migration phase routing. +- **Suspected surface**: **Both.** The routing gap is in modern code + (`com.dotcms.content.elasticsearch.business.ESContentFactoryImpl`, + `com.dotcms.content.index.PhaseRouter`), but the reason the failure becomes an empty `200` is + legacy swallow behavior in `com.dotcms.rendering.velocity.viewtools.content.util.ContentUtils` + and `com.dotcms.rest.ContentHelper`. +- **Related known decisions**: `docs/backend/OPENSEARCH_MIGRATION.md` lines 105, 402, 676 and 901 + all state the Phase 2 read fallback as designed behavior — the document is the contract being + violated. Phase 3 must keep propagating read failures (no fallback; Elasticsearch is + decommissioned). The plan formally consults `dotCMS/platform-adrs`. + +## Root-Cause Hypothesis + +Two independent layers. Both were verified in the current tree. + +### Layer 1 — the router is never called (the actual defect) + +`ESContentFactoryImpl.java:273` picks an engine with a bare ternary and calls the chosen +provider **directly**: + +```java +ContentFactoryIndexOperations indexOperationsDelegate(){ + return isMigrationComplete() || isReadEnabled() ? indexOperationsOS : indexOperationsES ; +} +``` + +Its five read call sites — `:1352` `search`, `:1607` `indexCount`, `:1616` `searchHits`, +`:1634` `indexSearchScroll`, `:1669` `createScrollQuery` — all invoke the selected provider with +no router in between. `PhaseRouter.read` (`PhaseRouter.java:187-199`) and `readChecked` +(`:298-310`) implement the fallback correctly and are simply unreachable from this path; the +class contains no reference to `PhaseRouter` at all. + +The stack proves it by omission — no `PhaseRouter` frame between +`ESContentFactoryImpl.indexCount` and `ContentFactoryIndexOperationsOS`. The same outage through +`IndexAPIImpl` (the health check) *does* carry `PhaseRouter.read(PhaseRouter.java:192)`. Same +failure, two paths, one routed. + +### Layer 2 — why the same outage yields 200/total=0 for some types and 500 for others + +> This supersedes the explanation in the GitHub issue body, which attributes the `200/total=0` +> to the `OpenSearchException → ERROR_HIT` branch. That is **wrong** for the observed case: a +> `ConnectException` is not an `OpenSearchException`, so it never reaches that branch. The +> issue body needs correcting. + +1. `ContentHelper.pullContent:310` runs the **count first**. A `CountRequest` carries no + offset/limit, so varying the offset does not change the count cache key — content types + queried before the outage had a cached count and did not throw; types not previously queried + (Hero, ContactCard) missed the cache, the count threw, → **500**. +2. `ContentUtils.java:302` has a `catch (Throwable)` that logs a one-line-truncated `WARN` and + returns an empty list, swallowing the search failure entirely. +3. `ContentHelper.java:318` — `if (contentlets.isEmpty() && offset <= resultsSize) { resultsSize = 0; }` + overwrites the real cached count with `0` → **200 / `total=0`**. + +The `ERROR_HIT` branch (`ContentFactoryIndexOperationsOS.java:109-121`) is nonetheless a real +second-order hazard: it converts an `OpenSearchException` into a legitimate-looking empty result, +leaving the router nothing to catch. It is **inherited Elasticsearch behavior**, structurally +identical in `ContentFactoryIndexOperationsES.java:56` and `:151-161` — so any change to it must +be scoped carefully. Cache poisoning is bounded: `shouldQueryCache(exceptionMsg)` only caches +`ERROR_HIT` for `parse_exception` / `search_phase_execution_exception`, so a connection failure +is not cached. + +## Fix Scope & Non-Goals *(mandatory)* + +**In scope**: + +- Route the five read call sites of `ESContentFactoryImpl.indexOperationsDelegate()` through + `PhaseRouter.read` / `readChecked`, following the router pattern already documented in + `PhaseRouter`'s class Javadoc and used by `IndexAPIImpl`. +- Ensure each Phase 2 fallback occurrence is logged at `ERROR` (the router already does this; + confirm no `WARN`-only path remains upstream of it). +- Make the OpenSearch read path in Phase 2 surface a *throwable* failure to the router rather + than an empty result, for the failure classes the router is meant to catch — including a + missing OpenSearch counterpart index. +- Tests: Phase 2 falls back and logs `ERROR`; Phase 3 propagates; the query cache does not mask + a live failure. + +**Explicitly out of scope / non-goals**: + +- **No change to Elasticsearch behavior.** `ContentFactoryIndexOperationsES` keeps its current + `ERROR_HIT` semantics; Phase 0 and Phase 1 read behavior is untouched. +- **No fallback in Phase 3.** Failures must keep propagating there by design. +- **No broad rewrite of the legacy swallow.** `ContentUtils.java:302`'s `catch (Throwable)` and + `ContentHelper.java:318`'s `resultsSize = 0` overwrite are pre-existing and affect all phases, + including pure Elasticsearch. They explain the *symptom*, and fixing Layer 1 removes the + Phase 2 case. Changing them repo-wide is a separate, larger issue — file it, do not fold it in. +- **No `IndexAPI` generic parameterization** — that belongs in its own PR. +- No write-path changes. Write failure semantics per phase are settled elsewhere. +- No new configuration property or feature flag to switch the fallback on and off — it is + documented, unconditional Phase 2 behavior. + +## Regression Risk *(mandatory)* + +- **Blast radius**: `ESContentFactoryImpl`'s five read call sites are the funnel for essentially + all content search — `/api/content/_search`, `ContentletAPI.search`/`searchIndex`/`indexCount`, + Velocity `$dotcontent.pull`, URL maps, Site Search, scroll/pagination consumers, and the + admin UI content browser. Any behavior change here is felt site-wide in every phase, so the + routing change must be a pure pass-through in phases 0, 1 and 3. +- **Callers that depend on empty-instead-of-exception**: making the OpenSearch read throw where + it currently returns `ERROR_HIT` changes the contract for anything that today receives an + empty result. These callers must be enumerated before that change lands, and each confirmed + either unaffected or explicitly handled. If the enumeration turns out to be large, the routing + fix (Layer 1) still stands on its own and can ship without touching `ERROR_HIT`. +- **Scroll and pagination**: `indexSearchScroll` and `createScrollQuery` hold engine-specific + cursor state. A mid-scroll fallback cannot resume an OpenSearch scroll on Elasticsearch — the + plan must decide whether these two sites fall back at all or only propagate, and say so. +- **Backward compatibility**: no API contract, response shape, serialized state, DB schema or + index mapping changes. Not rollback-unsafe. +- **Data considerations**: none — no data repair needed. Content missing from a read during an + outage was never lost; it is in Elasticsearch the whole time. +- **Performance**: a fallback doubles the work for a failing read. Bounded to the outage window, + and the failing engine fails fast, but the plan should confirm no retry storm (one fallback + attempt per read, no nesting). + +## Acceptance & Verification *(mandatory)* + +- **AC-001**: The reproduction above produces the expected behavior — in Phase 2 with + OpenSearch stopped and Elasticsearch healthy, `POST /api/content/_search` returns the same + non-zero result set Elasticsearch holds for **every** content type that has live content. No + `total=0`, no 500. +- **AC-002**: All five read call sites of `indexOperationsDelegate()` (`:1352`, `:1607`, `:1616`, + `:1634`, `:1669`) go through `PhaseRouter.read` / `readChecked` instead of invoking the + selected provider directly — or, for a site the plan excludes (see scroll/pagination above), + the exclusion is documented in code with its reason. +- **AC-003**: Each fallback occurrence is logged at `ERROR`, once per read, naming the failing + OpenSearch operation and the cause — so an outage is visible to log-based monitoring per the + design's early-warning signal. No `WARN`-only swallow remains on the Phase 2 fallback path. +- **AC-004**: Phase 0, 1 and 3 read behavior is byte-for-byte unchanged: 0/1 read from + Elasticsearch with no fallback; 3 reads from OpenSearch and propagates failures. Verified by + test, not by inspection. +- **AC-005**: A missing OpenSearch counterpart index in Phase 2 is served from Elasticsearch + rather than returning empty, matching `OPENSEARCH_MIGRATION.md:402` — this also covers the + reactivated-backup-index scenario. +- **AC-006**: If the `OpenSearchException`/`ERROR_HIT` branch is changed, Elasticsearch's + equivalent branch (`ContentFactoryIndexOperationsES.java:151-161`) is **not** changed, Phase 3 + still propagates, and the enumeration of callers relying on the empty result is recorded in + the plan with each one confirmed unaffected. +- **AC-007**: A repeated identical query does not mask a live OpenSearch failure by serving a + stale pre-outage result from the query cache — the caller either gets fallback results or a + surfaced error, never a silent stale hit presented as current. + +**Verification method**: + +- Integration test — Phase 2 with the OpenSearch provider stubbed to throw: a content search + returns the Elasticsearch result set and logs at `ERROR`. +- Integration test — the same stubbed failure in Phase 3 propagates and does **not** fall back. +- Integration test — phases 0/1 unchanged (AC-004). +- Integration test — the query-cache path (AC-007). +- All of the above live in `dotcms-integration`, are named `*Test` (not `*IT`), and are + **registered in the matching `@SuiteClasses` suite** — an unregistered class compiles, passes + CI green, and never runs. +- Run: `./mvnw verify -pl :dotcms-integration -Dcoreit.test.skip=false -Dit.test=` + (requires the WAR to be installed first, not just compiled). +- Manual re-run of the reproduction on the Phase 2 real-data rig, varying `offset` per call. + +## Assumptions + +- The intended behavior is what `docs/backend/OPENSEARCH_MIGRATION.md` states; no ADR overrides + it. The plan will confirm against `dotCMS/platform-adrs`. +- Elasticsearch being in sync in Phase 2 is a given — Phase 2 is dual-write, so ES is + continuously written. A fallback returning stale-but-present ES data is strictly better than + returning nothing. +- The scroll/pagination question (AC-002 exclusion) is a plan-phase decision, not a + specification gap: either answer satisfies the design as long as it is deliberate and + documented. +- The GitHub issue body will be corrected to match Layer 2 above before implementation, so the + issue and this spec do not disagree on root cause.