dotCMS has two AI surfaces and they are under different pressures. {@code /api/v1/ai/*} + * evolves when dotCMS decides it should. This family is pinned to an external standard that + * nobody here controls, so it changes when that standard changes — on someone else's schedule. + * Keeping its request and response semantics in a class of their own is what lets that happen + * without editing a line that a shipped endpoint depends on.
+ * + *What deliberately did not move here is model construction, caching and the + * per-site fallback chain. Those stay in {@link LangChain4jAIClient} because + * {@code LangChain4jAIClient.flushCachesForHost(String)} is what evicts a site's cached providers + * when its credentials are rotated, and {@code AIAppListener} is wired to that one class. A second + * client keeping its own cache would go on serving a revoked key until its TTL expired, with + * nothing to notice. So this class borrows models through + * {@link LangChain4jAIClient#withChatModel}, {@link LangChain4jAIClient#withStreamingChatModel}, + * {@link LangChain4jAIClient#withEmbeddingModel} and {@link LangChain4jAIClient#withImageModel}, + * and owns no state of its own.
+ * + *Those accessors hand over the name of the model that actually served, which after a fallback + * hop is not the one that was asked for. That name — not {@link InferenceRequest#model()} — is + * what {@link InferenceResponse#model()} reports, so a caller can tell a fallback happened. For + * the same reason the requested model is never set on the outgoing {@link ChatRequest}: the site's + * configuration, not the caller, decides which provider model runs.
+ * + *Nothing here logs message content, tool arguments or provider error envelopes: they carry + * customer data, and an error event is given a description written here rather than whatever the + * provider echoed back.
+ */ +public final class InferenceAIClient { + + private static final LazyThe model is borrowed from {@link LangChain4jAIClient}, so a site's fallback chain applies + * and the returned {@link InferenceResponse#model()} names the model that actually served. + * Usage is reported only when the provider reported it; when it did not, the response carries + * {@link InferenceUsage#UNREPORTED} rather than counts invented here.
+ * + * @param appConfig the resolved site's configuration + * @param request the completion to run + * @return the assistant's turn, which may ask for tools instead of carrying text + * @throws RuntimeException if every model in the site's chain failed; the exception is the last + * failure, left for the REST layer to turn into a status + */ + public InferenceResponse complete(final AppConfig appConfig, final InferenceRequest request) { + final ChatRequest chatRequest = toChatRequest(request); + return LangChain4jAIClient.get().withChatModel( + appConfig, + (model, servingModel) -> toInferenceResponse(model.chat(chatRequest), servingModel)); + } + + /** + * Runs one completion and hands each event to {@code sink} as it arrives. + * + *The sink sees content fragments, tool-call fragments, and exactly one terminal event: a + * {@link InferenceStreamEvent.Finish} when generation ended, or an + * {@link InferenceStreamEvent.Error} when it did not. Never both — after the first event is + * written the HTTP status is already gone, so which variant ends the stream is the only way a + * caller can tell a failed answer from a complete one. A + * {@link InferenceStreamEvent.Usage} event follows the finish only when the caller set + * {@link InferenceRequest#includeUsageInStream()}; sending it unasked breaks readers that + * reject the empty choices array it serializes to.
+ * + *{@link InferenceLimits#completionTimeoutSeconds()} is a hard ceiling: a provider that + * stops producing events ends the stream with an error rather than parking the calling thread + * indefinitely. Failures are reported through the sink rather than thrown, because once + * streaming has begun the sink is the only channel left.
+ * + * @param appConfig the resolved site's configuration + * @param request the completion to run + * @param sink receives every event, in order + */ + public void stream(final AppConfig appConfig, + final InferenceRequest request, + final ConsumerOne call rather than one per input, because the batch is what the caller sent and what + * the provider bills as a unit: embedding them separately would report usage that describes + * none of them and would multiply the site's spend on a request it was never asked to split. + * The vectors come back in the order the inputs were given, which is what lets the caller + * stamp each entry with the index of the text it embedded.
+ * + *Nothing here logs the input: it is customer content by definition.
+ * + * @param appConfig the resolved site's configuration + * @param inputs the texts to embed, in the order the caller sent them + * @return the vectors, the model that served, and the usage for the whole batch + * @throws RuntimeException if every model in the site's chain failed; the exception is the last + * failure, left for the REST layer to turn into a status + */ + public EmbeddingBatch embed(final AppConfig appConfig, final List{@link LangChain4jAIClient#withImageModel} already asks the provider for the inline form, + * so for most providers the bytes arrive inline and nothing further is needed. A provider that + * has no such option answers with a URL regardless; that URL is fetched and re-encoded here + * rather than refused, because the upstream artifact exists either way at that point and + * declining would break image generation on part of a multi-provider gateway to avoid an + * exposure already incurred. What the guarantee actually covers is the caller: they never + * receive an addressable artifact.
+ * + *A request for a single image goes through {@code generate(prompt)} rather than through + * {@code generate(prompt, 1)}. The multi-image overload is a {@code default} method on + * {@link ImageModel} that throws unless a provider overrides it, so routing the common case + * through it would break every provider that has not — for a count where the two calls mean + * exactly the same thing.
+ * + *Neither the prompt nor the provider's URL is logged; the first is customer content and + * the second addresses content generated from it.
+ * + * @param appConfig the resolved site's configuration + * @param prompt what to generate + * @param size the size the caller asked for, or null to use the site's configured one + * @param count how many images to generate; at least one + * @return the images as base64, and the model that served + * @throws RuntimeException if every model in the site's chain failed, if the provider cannot + * generate several images at once, or if it produced nothing that + * could be turned into bytes + */ + public GeneratedImages generateImages(final AppConfig appConfig, + final String prompt, + final String size, + final int count) { + return LangChain4jAIClient.get().withImageModel(appConfig, size, (model, servingModel) -> { + if (count > 1 && !supportsMultipleImages(model)) { + throw new MultipleImagesUnsupportedException(servingModel); + } + final ListThe multi-image call is a default method on the provider abstraction that throws unless + * the implementation overrides it, so declaring the image capability says nothing about + * whether this particular provider honours a count. Asking the class which one it inherited is + * exact: it needs no list of provider names to be kept current, and it cannot be fooled by a + * message string that the library is free to change.
+ * + *Probing beforehand rather than catching afterwards matters because the thrown type is + * {@code IllegalArgumentException} — indistinguishable from a genuine complaint about the + * arguments, and so not safe to translate on sight.
+ * + * @param model the image model handed over by the accessor + * @return whether the multi-image call is implemented rather than inherited + */ + private static boolean supportsMultipleImages(final ImageModel model) { + try { + return model.getClass() + .getMethod("generate", String.class, int.class) + .getDeclaringClass() != ImageModel.class; + } catch (final NoSuchMethodException e) { + return false; + } + } + + /** + * Drives one streaming exchange to its terminal event. + * + *Nothing is thrown out of here once an event has been emitted: the fallback chain in + * {@link LangChain4jAIClient} retries on a thrown exception, and a retry after the sink has + * seen part of an answer would splice two streams together. Before the first event there is + * nothing to splice, so a failure there is rethrown and the next model gets its turn.
+ * + * @param model the streaming model handed over by the accessor + * @param chatRequest the mapped request + * @param state collects events and guarantees a single terminal one + */ + private void streamWithModel(final StreamingChatModel model, + final ChatRequest chatRequest, + final StreamState state) { + final int timeoutSeconds = InferenceLimits.current().completionTimeoutSeconds(); + try { + model.chat(chatRequest, new StreamingChatResponseHandler() { + + @Override + public void onPartialResponse(final String token) { + state.contentDelta(token); + } + + @Override + public void onPartialToolCall(final PartialToolCall partialToolCall) { + state.partialToolCall(partialToolCall); + } + + @Override + public void onCompleteToolCall(final CompleteToolCall completeToolCall) { + state.completeToolCall(completeToolCall); + } + + @Override + public void onCompleteResponse(final ChatResponse response) { + state.finish(response); + } + + @Override + public void onError(final Throwable throwable) { + Logger.warn(InferenceAIClient.class, + "Inference stream failed: " + throwable.getClass().getSimpleName()); + state.fail(toInferenceError(throwable)); + } + }); + } catch (final RuntimeException e) { + if (!state.hasEmitted()) { + throw e; + } + Logger.warn(InferenceAIClient.class, + "Inference stream aborted: " + e.getClass().getSimpleName()); + state.fail(toInferenceError(e)); + return; + } + + if (!state.awaitTerminal(timeoutSeconds)) { + Logger.warn(InferenceAIClient.class, + "Inference stream exceeded the " + timeoutSeconds + " second ceiling"); + state.fail(InferenceError.upstream( + "The completion exceeded the configured ceiling of " + timeoutSeconds + " seconds")); + } + } + + /** + * Maps a completion request into the provider-facing request. + * + *The requested model is not mapped: the site's configuration chooses which provider model + * runs, and overriding it here would defeat the fallback chain the accessors apply.
+ * + * @param request the completion to run + * @return the request to hand a chat model + */ + private static ChatRequest toChatRequest(final InferenceRequest request) { + final ChatRequest.Builder builder = ChatRequest.builder() + .messages(toChatMessages(request.messages())) + .temperature(request.temperature()) + .maxOutputTokens(request.maxOutputTokens()) + .topP(request.topP()); + + if (!request.stopSequences().isEmpty()) { + builder.stopSequences(request.stopSequences()); + } + if (request.hasTools()) { + builder.toolSpecifications(toToolSpecifications(request.tools(), request.toolChoice())); + } + if (request.toolChoice() != null) { + builder.toolChoice(toToolChoice(request.toolChoice())); + } + if (request.responseFormat() != null) { + builder.responseFormat(toResponseFormat(request.responseFormat())); + } + return builder.build(); + } + + /** + * Maps the conversation, turn by turn, preserving order and tool-call identity. + * + * @param messages the conversation, oldest first + * @return the provider-facing messages + */ + private static ListWhen the caller forced one named tool, only that tool is offered. LangChain4J's shared + * chat API has no "call this exact tool" choice — its strongest is "call some tool" — so + * narrowing the offer is what turns that into the caller's actual instruction. If the forced + * name matches nothing declared, every tool is offered and the provider is left to reject it, + * which is a clearer failure than silently sending no tools at all.
+ * + * @param tools the declared tools + * @param toolChoice the caller's preference, possibly null + * @return the provider-facing tool specifications + */ + private static List{@code FUNCTION} becomes {@code REQUIRED}, paired with the narrowed tool list built by + * {@link #toToolSpecifications}; together they say what the caller meant.
+ * + * @param toolChoice the caller's preference + * @return the provider-facing choice + */ + private static dev.langchain4j.model.chat.request.ToolChoice toToolChoice( + final com.dotcms.inference.model.ToolChoice toolChoice) { + switch (toolChoice.mode()) { + case NONE: + return dev.langchain4j.model.chat.request.ToolChoice.NONE; + case REQUIRED: + case FUNCTION: + return dev.langchain4j.model.chat.request.ToolChoice.REQUIRED; + case AUTO: + default: + return dev.langchain4j.model.chat.request.ToolChoice.AUTO; + } + } + + /** + * Maps the requested output shape. + * + * @param responseFormat the caller's requested shape + * @return the provider-facing response format + */ + private static dev.langchain4j.model.chat.request.ResponseFormat toResponseFormat( + final com.dotcms.inference.model.ResponseFormat responseFormat) { + switch (responseFormat.type()) { + case JSON_OBJECT: + return dev.langchain4j.model.chat.request.ResponseFormat.JSON; + case JSON_SCHEMA: + return dev.langchain4j.model.chat.request.ResponseFormat.builder() + .type(ResponseFormatType.JSON) + .jsonSchema(JsonSchema.builder() + .name(RESPONSE_SCHEMA_NAME) + .rootElement(toObjectSchema(responseFormat.schema())) + .build()) + .build(); + case TEXT: + default: + return dev.langchain4j.model.chat.request.ResponseFormat.TEXT; + } + } + + /** + * Wraps a caller-supplied JSON Schema in the object schema LangChain4J expects. + * + *Only the envelope — properties, required, description, additionalProperties, definitions — + * is read. Each property's schema is carried through verbatim as a raw element, because dotCMS + * neither interprets nor validates it: the schema is the caller's contract with their own tool, + * and re-modelling it here would quietly drop the keywords this mapping does not know about.
+ * + * @param schema the caller's JSON Schema document + * @return the equivalent object schema + */ + private static JsonObjectSchema toObjectSchema(final JsonNode schema) { + final JsonObjectSchema.Builder builder = JsonObjectSchema.builder(); + + final JsonNode description = schema.get(SCHEMA_DESCRIPTION); + if (description != null && description.isTextual()) { + builder.description(description.asText()); + } + + final MapA provider that reports nothing is read from what it returned instead: an assistant turn + * asking for tools stopped to have them executed, whatever the provider forgot to say.
+ * + * @param finishReason the provider's reason, possibly null + * @param aiMessage the assistant turn, used when the provider was silent + * @return the reason to report + */ + private static com.dotcms.inference.model.FinishReason toFinishReason( + final dev.langchain4j.model.output.FinishReason finishReason, + final AiMessage aiMessage) { + if (finishReason == null) { + return aiMessage != null && aiMessage.hasToolExecutionRequests() + ? com.dotcms.inference.model.FinishReason.TOOL_CALLS + : com.dotcms.inference.model.FinishReason.STOP; + } + switch (finishReason) { + case LENGTH: + return com.dotcms.inference.model.FinishReason.LENGTH; + case TOOL_EXECUTION: + return com.dotcms.inference.model.FinishReason.TOOL_CALLS; + case CONTENT_FILTER: + return com.dotcms.inference.model.FinishReason.CONTENT_FILTER; + case STOP: + case OTHER: + default: + return com.dotcms.inference.model.FinishReason.STOP; + } + } + + /** + * Maps reported token counts. + * + * @param tokenUsage the provider's counts, possibly null or partly absent + * @return the counts, or {@link InferenceUsage#UNREPORTED} when the provider reported none + */ + private static InferenceUsage toInferenceUsage(final TokenUsage tokenUsage) { + if (tokenUsage == null) { + return InferenceUsage.UNREPORTED; + } + final InferenceUsage usage = new InferenceUsage( + tokenUsage.inputTokenCount(), + tokenUsage.outputTokenCount(), + tokenUsage.totalTokenCount()); + return usage.isReported() ? usage : InferenceUsage.UNREPORTED; + } + + /** + * Turns a failure into an error safe to hand a client. + * + *A provider's own message is never passed through: it can quote the prompt back. Messages + * dotCMS wrote — a misconfigured site, for instance — are safe and are kept, because they name + * something the caller or an operator can actually fix.
+ * + * @param throwable what went wrong + * @return the error to report + */ + private static InferenceError toInferenceError(final Throwable throwable) { + if (throwable instanceof IllegalArgumentException) { + return InferenceError.invalidRequest(nullToEmpty(throwable.getMessage()).isBlank() + ? "The request could not be served with this site's configuration" + : throwable.getMessage(), null); + } + return InferenceError.upstream(UPSTREAM_FAILURE_MESSAGE); + } + + /** + * @param value a possibly null string + * @return the value, or an empty string + */ + private static String nullToEmpty(final String value) { + return value == null ? "" : value; + } + + /** + * Reads an image as base64, fetching it first when the provider only gave a link to it. + * + * @param image the provider's image + * @return the image bytes, base64 encoded + */ + private static String toBase64(final Image image) { + if (image == null) { + throw new IllegalStateException(NO_IMAGE_MESSAGE); + } + if (image.base64Data() != null && !image.base64Data().isBlank()) { + return image.base64Data(); + } + if (image.url() == null) { + throw new IllegalStateException(NO_IMAGE_MESSAGE); + } + return fetchAndEncode(image.url()); + } + + /** + * Downloads a provider-hosted image and encodes it. + * + *Bounded by {@link #IMAGE_FETCH_TIMEOUT_SECONDS} so a provider whose CDN hangs cannot park + * the request thread for as long as it likes. The address is the provider's own, taken from a + * response to a request dotCMS made to an endpoint the site configured, and it is never logged + * or handed back to the caller.
+ * + * @param url where the provider put the image + * @return the image bytes, base64 encoded + */ + private static String fetchAndEncode(final URI url) { + final Duration timeout = Duration.ofSeconds(IMAGE_FETCH_TIMEOUT_SECONDS); + try { + final HttpClient httpClient = HttpClient.newBuilder() + .connectTimeout(timeout) + .followRedirects(HttpClient.Redirect.NORMAL) + .build(); + final HttpResponseAn embeddings call generates nothing, so there is no completion count to report and the + * total equals the prompt count. A provider that reported only the prompt count is therefore + * read as having reported the total too — that is arithmetic on what it said, not an estimate + * of what it did not.
+ * + * @param tokenUsage the provider's counts, possibly null or partly absent + * @return the counts, or {@link InferenceUsage#UNREPORTED} when the provider reported none + */ + private static InferenceUsage toEmbeddingUsage(final TokenUsage tokenUsage) { + if (tokenUsage == null) { + return InferenceUsage.UNREPORTED; + } + final Integer inputTokens = tokenUsage.inputTokenCount(); + final Integer totalTokens = tokenUsage.totalTokenCount() == null + ? inputTokens + : tokenUsage.totalTokenCount(); + final InferenceUsage usage = new InferenceUsage(inputTokens, null, totalTokens); + return usage.isReported() ? usage : InferenceUsage.UNREPORTED; + } + + /** + * One batch of embeddings, as the provider produced them. + * + * @param model the model that actually served, after any fallback hop + * @param vectors one vector per input, in the order the inputs were given + * @param usage tokens consumed by the whole batch + */ + public record EmbeddingBatch(String model, ListProvider callbacks arrive on the provider's own threads, so every emission is serialized + * here and the terminal flag is checked under the same lock. That is also what keeps a timeout + * racing a late completion from producing both an error and a finish.
+ */ + private static final class StreamState { + + private final ConsumerProviders that do stream tool calls send this after the fragments, and re-emitting the + * arguments here would have a reader concatenate them twice into malformed JSON.
+ * + * @param completeToolCall the finished tool call + */ + private void completeToolCall(final CompleteToolCall completeToolCall) { + if (fragmentedToolCalls.contains(completeToolCall.index())) { + return; + } + final ToolExecutionRequest request = completeToolCall.toolExecutionRequest(); + emit(new InferenceStreamEvent.ToolCallDelta( + completeToolCall.index(), request.id(), request.name(), request.arguments())); + } + + /** + * Ends the stream normally, with the usage event only if the caller asked for one. + * + * @param response the provider's finished response + */ + private synchronized void finish(final ChatResponse response) { + if (terminated.get()) { + return; + } + final AiMessage aiMessage = response == null ? null : response.aiMessage(); + emit(new InferenceStreamEvent.Finish(toFinishReason( + response == null ? null : response.finishReason(), aiMessage))); + if (includeUsage) { + emit(new InferenceStreamEvent.Usage( + toInferenceUsage(response == null ? null : response.tokenUsage()))); + } + terminate(); + } + + /** + * Ends the stream as failed. Ignored once the stream has already ended, so a finish and an + * error can never both be reported. + * + * @param error what went wrong + */ + private synchronized void fail(final InferenceError error) { + if (terminated.get()) { + return; + } + emit(new InferenceStreamEvent.Error(error)); + terminate(); + } + + /** + * Hands one event to the sink. + * + *A sink that throws is a caller that has gone away — a disconnected client, typically. + * There is nowhere left to report anything, so the stream is ended silently and the waiting + * thread released rather than held until the timeout.
+ * + * @param event the event to emit + */ + private synchronized void emit(final InferenceStreamEvent event) { + if (terminated.get()) { + return; + } + try { + sink.accept(event); + emitted.set(true); + } catch (final RuntimeException e) { + Logger.warn(InferenceAIClient.class, + "Inference stream consumer rejected an event, ending the stream: " + + e.getClass().getSimpleName()); + terminate(); + } + } + + /** Marks the stream ended and releases whoever is waiting on it. */ + private void terminate() { + terminated.set(true); + terminalLatch.countDown(); + } + } +} diff --git a/dotCMS/src/main/java/com/dotcms/ai/client/langchain4j/LangChain4jAIClient.java b/dotCMS/src/main/java/com/dotcms/ai/client/langchain4j/LangChain4jAIClient.java index 7abab2c6a0cc..6b0d33b46ed8 100644 --- a/dotCMS/src/main/java/com/dotcms/ai/client/langchain4j/LangChain4jAIClient.java +++ b/dotCMS/src/main/java/com/dotcms/ai/client/langchain4j/LangChain4jAIClient.java @@ -45,6 +45,8 @@ import java.nio.charset.StandardCharsets; import java.util.ArrayList; import java.util.List; +import java.util.function.BiConsumer; +import java.util.function.BiFunction; import java.util.function.Function; import java.util.concurrent.CountDownLatch; import java.util.concurrent.TimeUnit; @@ -72,6 +74,22 @@ public class LangChain4jAIClient implements AIClient { private static final ObjectMapper MAPPER = DotObjectMapperProvider.createDefaultMapper(); private static final long MODEL_CACHE_TTL_HOURS = 1; private static final long STREAMING_TIMEOUT_SECONDS = 300; + private static final String CHAT_SECTION = "chat"; + private static final String EMBEDDINGS_SECTION = "embeddings"; + private static final String IMAGE_SECTION = "image"; + + /** + * Response format asked of an image provider by the {@code /api/inference/v1} family, so the + * bytes arrive inline and no separately-addressable artifact is minted upstream. + */ + private static final String INLINE_IMAGE_FORMAT = "b64_json"; + + /** + * Cache-key discriminator for models the inference family borrows. Those differ from the ones + * the legacy endpoints get — they are built asking for the inline image form — so they cannot + * share a cache entry with them even for the same site, model and size. + */ + private static final String INFERENCE_KEY_SEGMENT = ":inference"; private final CacheExists so the {@code /api/inference/v1} family can own its own request and response + * semantics — which are dictated by an external standard and will change when that standard + * changes — without owning model construction, caching or eviction. Those stay here, in one + * place, for one reason: {@link #flushCachesForHost(String)} is what evicts a site's cached + * providers when its credentials are rotated, and it is wired to this class alone + * ({@code AIAppListener}). A second client holding its own cache would keep serving a revoked + * key until the TTL expired, with no symptom to notice.
+ * + *The executor receives the model's name alongside the model because a fallback hop means + * the model that served is not the model that was asked for, and a caller reporting back to a + * client has to say which one actually ran.
+ * + * @param appConfig the resolved site's configuration + * @param executor receives a chat model and its name, and produces the result + * @paramThe streaming counterpart of {@link #withChatModel}; see that method for why model + * acquisition stays in this class rather than moving to the caller.
+ * + * @param appConfig the resolved site's configuration + * @param executor receives a streaming chat model and its name + */ + public void withStreamingChatModel(final AppConfig appConfig, + final BiConsumerThe embeddings counterpart of {@link #withChatModel}; see that method for why model + * acquisition stays in this class rather than moving to the caller. The model is built from + * the site's {@code embeddings} section, which a site configures independently of its chat + * models.
+ * + * @param appConfig the resolved site's configuration + * @param executor receives an embedding model and its name, and produces the result + * @paramThe image counterpart of {@link #withChatModel}, with two differences that are the + * caller's request rather than the site's configuration. The requested {@code size} is applied + * over the configured one, because a caller who named a size changed both what they receive + * and what the site pays. And {@code responseFormat} is set to {@value #INLINE_IMAGE_FORMAT} + * so a provider that offers the choice never mints a hosted artifact in the first place; + * providers that ignore it still answer, and the caller re-encodes what comes back.
+ * + *Both are folded into the cache key, so a model asked for one size is never handed to a + * request that asked for another, and the legacy image endpoint — which wants the provider's + * own default format — never receives one of these.
+ * + * @param appConfig the resolved site's configuration + * @param size the size the caller asked for, or null/blank to use the configured one + * @param executor receives an image model and its name, and produces the result + * @paramExists so that the model gate every {@code /api/inference/v1} endpoint applies, and the + * listing that tells a caller what will pass it, read the configuration through the class that + * owns it rather than each re-implementing the same parse. Fallback chains are returned whole + * and in configured order, because every entry is a name the gate accepts.
+ * + *A site with no usable configuration for that section yields an empty list rather than an + * exception. From where a caller stands, a model nobody configured and a section nobody + * configured are the same absence, and distinguishing them would disclose which sites have + * dotAI set up.
+ * + * @param appConfig the resolved site's configuration + * @param section the {@code providerConfig} section, e.g. {@code chat} + * @return the configured model names in fallback order; empty when there are none + */ + public ListDerived from the configuration rather than from the request, which is what makes two + * sites with different providers get separate model instances for free, and what makes a + * credential rotation change the key so the old instance is no longer reachable.
+ * + * @param appConfig the resolved site's configuration + * @return the prefix shared by every cache entry for that site and configuration + */ + private static String cacheKeyPrefix(final AppConfig appConfig) { + return appConfig.getHost() + ":" + appConfig.getProviderConfigHash(); + } + @Override public AIProvider getProvider() { return AIProvider.LANGCHAIN4J; @@ -336,6 +522,36 @@Generalised from the String-returning variant above, which now delegates here, so the + * fallback chain, the cache keying and the per-attempt logging have exactly one + * implementation. The executor additionally receives the name of the model it was handed, + * because a caller reporting results back to a client needs to say which model actually + * served — after a fallback hop that is not the one the caller asked for.
+ * + * @param cacheKeyPrefix the site-and-config-derived cache key prefix + * @param section the providerConfig section, e.g. {@code chat} + * @param baseConfig the parsed section config + * @param modelCache the cache for this model type + * @param modelBuilder builds a model from a config naming one model + * @param executor receives the model and the model's name, and produces the result + * @param{@code /api/inference/v1/images/generations} sets {@code b64_json} because FR-012 wants + * no hosted artifact minted upstream at all, not merely none returned to the caller. The + * legacy {@code /api/v1/ai/image} endpoint leaves it unset and keeps the provider's default, + * because its own response contract is a URL.
+ */ + @Nullable String responseFormat(); @Nullable Integer dimensions(); @Nullable String endpoint(); @Nullable String deploymentName(); diff --git a/dotCMS/src/main/java/com/dotcms/ai/rest/AiHostResolver.java b/dotCMS/src/main/java/com/dotcms/ai/rest/AiHostResolver.java index 492827d8a318..0ec0cf64f443 100644 --- a/dotCMS/src/main/java/com/dotcms/ai/rest/AiHostResolver.java +++ b/dotCMS/src/main/java/com/dotcms/ai/rest/AiHostResolver.java @@ -1,5 +1,7 @@ package com.dotcms.ai.rest; +import com.dotcms.ai.app.AppConfig; +import com.dotcms.ai.app.ConfigService; import com.dotmarketing.beans.Host; import com.dotmarketing.business.APILocator; import com.dotmarketing.business.web.WebAPILocator; @@ -9,6 +11,7 @@ import org.apache.commons.lang3.StringUtils; import javax.servlet.http.HttpServletRequest; +import java.util.Optional; /** * Resolves the target {@link Host} for a dotAI REST request from an optional {@code siteId} @@ -16,11 +19,120 @@ * absent or unresolvable. Shared by every dotAI endpoint that reads or tests a per-site * {@code providerConfig} ({@link CompletionsResource}, {@link AiProviderResource}). */ -final class AiHostResolver { +public final class AiHostResolver { private AiHostResolver() { } + /** + * Resolves the caller, the target site and that site's dotAI configuration in one step. + * + *This is the single entry point the {@code /api/inference/v1} family uses to obtain an + * {@link AppConfig}; resolving a host and fetching a configuration separately is what allowed + * two shipped dotAI endpoints to drift to opposite policies, and returning the three together + * removes the opportunity.
+ * + *Resolution is deliberately standard. When {@code siteOverride} is blank + * the site comes from the request exactly as it does everywhere else in dotCMS, default-site + * fallback included, because the server-side callers this family exists for routinely present + * a host name that is no site alias — internal DNS, container service names, {@code + * localhost}. Refusing those would have broken the primary supported deployment. The risk the + * fallback carries is that nobody can tell which site paid, so a fallback is logged here and + * the resolved site is reported back on every response.
+ * + *An explicit override is the one path that enforces site READ: {@link #findHost} resolves + * it as the caller, so a site they cannot see raises {@link DotSecurityException}.
+ * + * @param request the inbound request + * @param siteOverride an explicit site id or host name, or null/blank to resolve from the request + * @param user the authenticated caller + * @return the caller, the resolved site and its configuration + * @throws DotSecurityException when an explicit override names a site the caller cannot read + * @throws IllegalArgumentException when an explicit override names no known site + */ + public static ResolvedAiContext resolve(final HttpServletRequest request, + final String siteOverride, + final User user) throws DotSecurityException { + final Host host = StringUtils.isNotBlank(siteOverride) + ? requireOverriddenHost(siteOverride, user) + : resolveFromRequest(request); + return new ResolvedAiContext(user, host, ConfigService.INSTANCE.config(host)); + } + + /** + * Resolves an explicitly requested site, as the caller, so permissions apply. + * + * @param siteOverride the site id or host name the caller asked for + * @param user the authenticated caller + * @return the site + * @throws DotSecurityException when the caller cannot read it + */ + private static Host requireOverriddenHost(final String siteOverride, final User user) + throws DotSecurityException { + try { + final Host found = findHost(siteOverride, user); + if (found == null) { + throw new IllegalArgumentException("Site not found: " + sanitize(siteOverride)); + } + return found; + } catch (final DotSecurityException | IllegalArgumentException e) { + throw e; + } catch (final Exception e) { + throw new IllegalArgumentException("Site not found: " + sanitize(siteOverride), e); + } + } + + /** + * Resolves the site from the request, noting when the default site had to stand in. + * + *The fallback itself is standard dotCMS behaviour and is kept. What is not kept is its + * silence: an unmatched host name means some site's credentials are about to be spent on a + * request that did not name it, and that should be visible in the log as well as in the + * response.
+ * + * @param request the inbound request + * @return the resolved site + */ + private static Host resolveFromRequest(final HttpServletRequest request) { + final String serverName = request.getServerName(); + try { + final OptionalDeliberately not {@code getCurrentHostNoThrow}, which is the obvious call and the wrong + * one: it reads the {@code host_id} and {@code Host} request parameters before it ever looks + * at the server name (`HostWebAPIImpl.getCurrentHostFromRequest`), and FR-025 requires those + * be ignored on this family. They were already ignored whenever the server name matched a + * site, because that path returns before reaching here — so the effect was that a legacy + * override was honoured in precisely the case where it could redirect which site's + * credentials get spent, and ignored everywhere it was harmless.
+ * + * @return the default site + */ + private static Host defaultHost() { + try { + return APILocator.getHostAPI().findDefaultHost(APILocator.systemUser(), false); + } catch (final Exception e) { + throw new IllegalStateException("Could not resolve the default site: " + e.getMessage(), e); + } + } + /** * Resolves a host from {@code siteId} and falls back to the HTTP host on failure. * Throws {@link DotSecurityException} when the user lacks permission for the requested site. @@ -68,7 +180,7 @@ static Host resolveHostStrict(final String siteId, } } - static String sanitize(final String value) { + public static String sanitize(final String value) { return value == null ? "null" : value.replaceAll("[\r\n\t]", "_"); } diff --git a/dotCMS/src/main/java/com/dotcms/ai/rest/ResolvedAiContext.java b/dotCMS/src/main/java/com/dotcms/ai/rest/ResolvedAiContext.java new file mode 100644 index 000000000000..8cbc96a8861c --- /dev/null +++ b/dotCMS/src/main/java/com/dotcms/ai/rest/ResolvedAiContext.java @@ -0,0 +1,51 @@ +package com.dotcms.ai.rest; + +import com.dotcms.ai.app.AppConfig; +import com.dotmarketing.beans.Host; +import com.liferay.portal.model.User; + +/** + * The caller, the site their request resolved to, and that site's dotAI configuration — resolved + * together, in one step. + * + *The three are returned as a unit on purpose. Obtaining them separately is what lets a caller + * authorize against one site and then fetch configuration for another, which is not a theoretical + * risk: two shipped dotAI endpoints have already drifted to opposite model-passthrough policies + * by each making the decision for themselves. A type that can only be produced by the shared + * resolver removes the opportunity.
+ * + * @param user the authenticated caller + * @param host the site the request resolved to, after standard dotCMS host resolution + * @param config that site's dotAI configuration, including any system-level inheritance + */ +public record ResolvedAiContext(User user, Host host, AppConfig config) { + + public ResolvedAiContext { + if (user == null) { + throw new IllegalArgumentException("ResolvedAiContext user is required"); + } + if (host == null) { + throw new IllegalArgumentException("ResolvedAiContext host is required"); + } + if (config == null) { + throw new IllegalArgumentException("ResolvedAiContext config is required"); + } + } + + /** + * The identity to report as having served — and paid for — the request. + * + * @return the resolved site's identifier + */ + public String servingSiteId() { + return host.getIdentifier(); + } + + /** + * @return whether the resolved site, or the system level it inherits from, has any dotAI + * configuration at all + */ + public boolean isConfigured() { + return config != null && config.isEnabled(); + } +} diff --git a/dotCMS/src/main/java/com/dotcms/inference/model/FinishReason.java b/dotCMS/src/main/java/com/dotcms/inference/model/FinishReason.java new file mode 100644 index 000000000000..fa5f4373adc5 --- /dev/null +++ b/dotCMS/src/main/java/com/dotcms/inference/model/FinishReason.java @@ -0,0 +1,23 @@ +package com.dotcms.inference.model; + +/** + * Why a completion stopped. + * + *{@link #ERROR} has no equivalent in the chat-completions wire format, which has no finish + * reason meaning "this failed". It exists here because a stream that fails after it has begun + * must be distinguishable from one that finished, and the serializer turns it into an error + * event rather than a finish reason.
+ */ +public enum FinishReason { + + /** The model finished naturally. */ + STOP, + /** The model hit a token ceiling. */ + LENGTH, + /** The model is asking for one or more tools to be executed. */ + TOOL_CALLS, + /** The provider suppressed the response. */ + CONTENT_FILTER, + /** The exchange failed. Never serialized as a finish reason. */ + ERROR +} diff --git a/dotCMS/src/main/java/com/dotcms/inference/model/InferenceError.java b/dotCMS/src/main/java/com/dotcms/inference/model/InferenceError.java new file mode 100644 index 000000000000..a24e642a6556 --- /dev/null +++ b/dotCMS/src/main/java/com/dotcms/inference/model/InferenceError.java @@ -0,0 +1,61 @@ +package com.dotcms.inference.model; + +import java.io.Serializable; + +/** + * A refusal, in the shape the REST layer serializes to the standard error object. + * + *{@code httpStatus} is carried here but is deliberately not part of the + * serialized body. Retryability is conveyed by the HTTP status — 429 and 5xx — because that is + * what a standard client's back-off keys off; the standard error shape has no retryable field + * and inventing one would make the payload stop deserializing into a client library's own + * types.
+ * + * @param type the error family, e.g. {@code invalid_request_error} + * @param message a safe description; never the provider's raw envelope + * @param param the offending field where one can be named, otherwise null + * @param httpStatus the status to respond with; not serialized into the body + */ +public record InferenceError(String type, String message, String param, int httpStatus) + implements Serializable { + + public InferenceError { + if (type == null || type.isBlank()) { + throw new IllegalArgumentException("InferenceError type is required"); + } + if (message == null || message.isBlank()) { + throw new IllegalArgumentException("InferenceError message is required"); + } + } + + /** + * @param message what was wrong + * @param param the offending field, or null + * @return a 400 invalid-request error + */ + public static InferenceError invalidRequest(final String message, final String param) { + return new InferenceError("invalid_request_error", message, param, 400); + } + + /** + * @param model the model the caller asked for + * @return a 404 error in the shape clients recognise as an unknown model + */ + public static InferenceError noSuchModel(final String model) { + return new InferenceError("invalid_request_error", + "The model '" + model + "' is not configured for this site", "model", 404); + } + + /** + * @param message what failed upstream + * @return a 502 error flagged retryable by its status + */ + public static InferenceError upstream(final String message) { + return new InferenceError("api_error", message, null, 502); + } + + /** @return whether a standard client should retry, as implied by the status */ + public boolean isRetryable() { + return httpStatus == 429 || httpStatus >= 500; + } +} diff --git a/dotCMS/src/main/java/com/dotcms/inference/model/InferenceLimits.java b/dotCMS/src/main/java/com/dotcms/inference/model/InferenceLimits.java new file mode 100644 index 000000000000..5b7f94a13576 --- /dev/null +++ b/dotCMS/src/main/java/com/dotcms/inference/model/InferenceLimits.java @@ -0,0 +1,79 @@ +package com.dotcms.inference.model; + +import com.dotmarketing.util.Config; + +/** + * The capacity ceilings this endpoint family enforces. + * + *All of them are configurable and all of them have defaults, because the risk they bound is + * real but its right size is deployment-specific. Streaming is the reason most of them exist: a + * streamed completion parks a request thread for the whole generation rather than for one round + * trip, so concurrency and elapsed time are the scarce resources here, not request rate.
+ * + *{@link #maxImagesPerRequest()} bounds a different resource. Image generation is priced per + * image, so a single accepted request can multiply a site's provider spend by whatever it asked + * for, and per-site spend quotas are out of scope for this work. The ceiling is what stops one + * request being an unbounded bill.
+ * + *Read through {@link #current()} at the point of use rather than cached in a field, so an + * operator changing a property does not have to restart the node to see it take effect.
+ * + * @param maxConcurrentStreams streaming completions allowed at once on this node + * @param completionTimeoutSeconds hard ceiling on a single completion + * @param maxRequestBytes largest request body accepted + * @param maxImagesPerRequest most images one generation request may ask for + */ +public record InferenceLimits(int maxConcurrentStreams, + int completionTimeoutSeconds, + int maxRequestBytes, + int maxImagesPerRequest) { + + /** Streaming completions allowed at once on this node. */ + public static final String MAX_CONCURRENT_STREAMS_KEY = "DOT_INFERENCE_MAX_CONCURRENT_STREAMS"; + /** Hard ceiling on a single completion, in seconds. */ + public static final String COMPLETION_TIMEOUT_SECONDS_KEY = "DOT_INFERENCE_COMPLETION_TIMEOUT_SECONDS"; + /** Largest request body accepted, in bytes. */ + public static final String MAX_REQUEST_BYTES_KEY = "DOT_INFERENCE_MAX_REQUEST_BYTES"; + /** Most images one generation request may ask for. */ + public static final String MAX_IMAGES_PER_REQUEST_KEY = "DOT_INFERENCE_MAX_IMAGES_PER_REQUEST"; + + /** Each stream holds a request thread for the life of a completion. */ + public static final int DEFAULT_MAX_CONCURRENT_STREAMS = 50; + /** Five minutes; long enough for a slow reasoning model, short enough to bound a hung stream. */ + public static final int DEFAULT_COMPLETION_TIMEOUT_SECONDS = 300; + /** 1 MiB; holds a long multi-turn conversation with tool results, and bounds parse cost. */ + public static final int DEFAULT_MAX_REQUEST_BYTES = 1024 * 1024; + /** + * Ten, which is the ceiling the adopted format itself documents for this field. Taking the + * number from the format rather than inventing one means a client written against the + * standard meets the same limit here that it already handles there. + */ + public static final int DEFAULT_MAX_IMAGES_PER_REQUEST = 10; + + /** + * @return the limits as currently configured on this node + */ + public static InferenceLimits current() { + return new InferenceLimits( + Config.getIntProperty(MAX_CONCURRENT_STREAMS_KEY, DEFAULT_MAX_CONCURRENT_STREAMS), + Config.getIntProperty(COMPLETION_TIMEOUT_SECONDS_KEY, DEFAULT_COMPLETION_TIMEOUT_SECONDS), + Config.getIntProperty(MAX_REQUEST_BYTES_KEY, DEFAULT_MAX_REQUEST_BYTES), + Config.getIntProperty(MAX_IMAGES_PER_REQUEST_KEY, DEFAULT_MAX_IMAGES_PER_REQUEST)); + } + + /** + * @param bytes the size of an incoming request body + * @return whether it exceeds {@link #maxRequestBytes()} + */ + public boolean exceedsMaxRequestBytes(final long bytes) { + return bytes > maxRequestBytes; + } + + /** + * @param count how many images a generation request asked for + * @return whether it exceeds {@link #maxImagesPerRequest()} + */ + public boolean exceedsMaxImagesPerRequest(final int count) { + return count > maxImagesPerRequest; + } +} diff --git a/dotCMS/src/main/java/com/dotcms/inference/model/InferenceMessage.java b/dotCMS/src/main/java/com/dotcms/inference/model/InferenceMessage.java new file mode 100644 index 000000000000..a7559a99716d --- /dev/null +++ b/dotCMS/src/main/java/com/dotcms/inference/model/InferenceMessage.java @@ -0,0 +1,86 @@ +package com.dotcms.inference.model; + +import java.io.Serializable; +import java.util.List; + +/** + * One turn in an inference conversation. + * + *An {@link Role#ASSISTANT} turn may carry {@code toolCalls} instead of {@code content} when + * the model is asking for tools to be executed. A {@link Role#TOOL} turn carries the result and + * must reference, through {@code toolCallId}, the identity of the call that requested it — + * correlation is by identity, never by position.
+ * + * @param role who authored the turn; required + * @param content the text of the turn; may be null on an assistant turn carrying only tool calls + * @param toolCalls tool calls requested by the model; empty unless the role is ASSISTANT + * @param toolCallId the call this turn answers; required when the role is TOOL + * @param name the tool that produced the result; optional, TOOL turns only + */ +public record InferenceMessage(Role role, + String content, + ListThis type is deliberately not a binding of any wire format. Parsing a client's payload into + * it, and serializing a response back out, are the REST layer's job; nothing here mirrors the + * structure of the JSON that arrives. The point is that tool-call identity, reasoning turns and + * message ordering survive in a form a second wire format could also be written from, instead of + * being flattened into whichever format happened to be implemented first.
+ * + *It is {@link Serializable} so it can travel as the payload of an
+ * {@code AIRequest
A provider given an orphaned tool turn fails in its own way, at its own layer, having + * already been paid for the round trip. Catching it here turns that into a validation error + * naming the offending identity.
+ * + * @param messages the conversation to check + */ + private static void requireToolResultsAreCorrelated(final List{@code model} is the model that actually served the request, which is not necessarily the + * one asked for: a site's fallback chain may have moved on to a later entry. Reporting the one + * that ran, rather than the one requested, is what lets a caller tell that a fallback + * happened.
+ * + * @param id identity for this response + * @param model the model that served it, after any fallback + * @param createdEpochSeconds when it was produced + * @param message the assistant turn; may carry tool calls instead of content + * @param finishReason why generation stopped + * @param usage tokens consumed, {@link InferenceUsage#UNREPORTED} if the provider was silent + */ +public record InferenceResponse(String id, + String model, + long createdEpochSeconds, + InferenceMessage message, + FinishReason finishReason, + InferenceUsage usage) implements Serializable { + + public InferenceResponse { + if (id == null || id.isBlank()) { + throw new IllegalArgumentException("InferenceResponse id is required"); + } + if (model == null || model.isBlank()) { + throw new IllegalArgumentException("InferenceResponse model is required"); + } + if (message == null) { + throw new IllegalArgumentException("InferenceResponse message is required"); + } + if (finishReason == null) { + throw new IllegalArgumentException("InferenceResponse finishReason is required"); + } + usage = usage == null ? InferenceUsage.UNREPORTED : usage; + } +} diff --git a/dotCMS/src/main/java/com/dotcms/inference/model/InferenceStreamEvent.java b/dotCMS/src/main/java/com/dotcms/inference/model/InferenceStreamEvent.java new file mode 100644 index 000000000000..c08b0929ecd5 --- /dev/null +++ b/dotCMS/src/main/java/com/dotcms/inference/model/InferenceStreamEvent.java @@ -0,0 +1,118 @@ +package com.dotcms.inference.model; + +import java.util.List; + +/** + * One event in a streamed completion. + * + *Sealed so the serializer is a total function: every variant must be handled, and adding a + * sixth cannot silently fall through to a default branch. That matters most for + * {@link Error @Error} — the difference between a stream that failed and a stream that finished + * is carried entirely by which variant ends it, and by the serializer refusing to write the + * terminal done marker after an error.
+ */ +public sealed interface InferenceStreamEvent + permits InferenceStreamEvent.ContentDelta, + InferenceStreamEvent.ToolCallDelta, + InferenceStreamEvent.Finish, + InferenceStreamEvent.Usage, + InferenceStreamEvent.Error { + + /** + * A fragment of the answer's text. + * + * @param text the fragment, as produced + */ + record ContentDelta(String text) implements InferenceStreamEvent { + public ContentDelta { + text = text == null ? "" : text; + } + } + + /** + * A fragment of one tool call. + * + *{@code id} and {@code name} arrive on the first fragment of a call and are absent + * afterwards; {@code partialArguments} accumulates across fragments. {@code index} identifies + * which call in the turn a fragment belongs to, and is the only thing that does so on + * continuation fragments.
+ * + * @param index which call in the turn this fragment belongs to + * @param id the call's identity, present on its first fragment only + * @param name the tool's name, present on its first fragment only + * @param partialArguments the argument text carried by this fragment + */ + record ToolCallDelta(int index, String id, String name, String partialArguments) + implements InferenceStreamEvent { + public ToolCallDelta { + partialArguments = partialArguments == null ? "" : partialArguments; + } + } + + /** + * Generation stopped normally. + * + * @param reason why it stopped; never {@link FinishReason#ERROR} + */ + record Finish(FinishReason reason) implements InferenceStreamEvent { + public Finish { + if (reason == null) { + throw new IllegalArgumentException("Finish reason is required"); + } + if (reason == FinishReason.ERROR) { + throw new IllegalArgumentException( + "A failed stream ends with an Error event, not a Finish with reason ERROR"); + } + } + } + + /** + * Tokens consumed by the exchange. + * + *Emitted only when the caller asked for it. The event it serializes to carries an empty + * choices array, which some stream readers do not tolerate, so sending it unasked would break + * clients this family exists to support.
+ * + * @param usage the counts + */ + record Usage(InferenceUsage usage) implements InferenceStreamEvent { + public Usage { + if (usage == null) { + throw new IllegalArgumentException("Usage event requires counts"); + } + } + } + + /** + * The stream failed after it had begun. + * + *Once the first event is written the HTTP status is already sent and cannot carry the + * failure, so this variant is the only way a caller learns the answer is not complete.
+ * + * @param error what went wrong + */ + record Error(InferenceError error) implements InferenceStreamEvent { + public Error { + if (error == null) { + throw new IllegalArgumentException("Error event requires an error"); + } + } + } + + /** + * Reassembles the complete arguments of one tool call from its fragments, in arrival order. + * + * @param events the events seen so far + * @param index the call to reassemble + * @return the concatenated argument text, empty if no fragment matched + */ + static String reassembleArguments(final ListThis type is the reason the internal representation exists. Identity is carried in + * {@code id}, which comes from the provider and is what a later {@link Role#TOOL} turn + * correlates against. It is never derived from {@code index}: the chat-completions streaming + * format happens to fragment tool calls and key the fragments by position, but that is a + * property of one serialization, and rebuilding identity from it would bake that format into + * the model. {@code index} is retained only so a serializer can reproduce it.
+ * + * @param id provider-assigned identity; required + * @param name the tool to execute; required + * @param arguments the model's arguments as JSON text, passed through unparsed + * @param index position within the assistant turn; serialization detail only + */ +public record InferenceToolCall(String id, String name, String arguments, int index) + implements Serializable { + + public InferenceToolCall { + if (id == null || id.isBlank()) { + throw new IllegalArgumentException("InferenceToolCall id is required"); + } + if (name == null || name.isBlank()) { + throw new IllegalArgumentException("InferenceToolCall name is required"); + } + arguments = arguments == null ? "" : arguments; + } + + /** + * Creates a tool call at the head position, for callers that do not track ordering. + * + * @param id provider-assigned identity + * @param name the tool to execute + * @param arguments the model's arguments as JSON text + * @return a tool call at index {@code 0} + */ + public static InferenceToolCall of(final String id, final String name, final String arguments) { + return new InferenceToolCall(id, name, arguments, 0); + } +} diff --git a/dotCMS/src/main/java/com/dotcms/inference/model/InferenceToolSpec.java b/dotCMS/src/main/java/com/dotcms/inference/model/InferenceToolSpec.java new file mode 100644 index 000000000000..23da66d716a8 --- /dev/null +++ b/dotCMS/src/main/java/com/dotcms/inference/model/InferenceToolSpec.java @@ -0,0 +1,31 @@ +package com.dotcms.inference.model; + +import com.fasterxml.jackson.databind.JsonNode; + +import java.io.Serializable; + +/** + * A tool the caller declares the model may ask to execute. + * + *{@code parameters} is a JSON Schema document describing the tool's arguments. It is held as + * a JSON tree rather than a bound type because dotCMS neither interprets nor validates it — the + * schema is the caller's contract with their own tool, and it is passed to the provider as + * given.
+ * + * @param name the tool's name, as the model will refer to it; required + * @param description what the tool does; optional but strongly advised, the model reads it + * @param parameters JSON Schema for the arguments; required + */ +public record InferenceToolSpec(String name, String description, JsonNode parameters) + implements Serializable { + + public InferenceToolSpec { + if (name == null || name.isBlank()) { + throw new IllegalArgumentException("InferenceToolSpec name is required"); + } + if (parameters == null) { + throw new IllegalArgumentException( + "InferenceToolSpec parameters is required; use an empty object schema for a tool taking none"); + } + } +} diff --git a/dotCMS/src/main/java/com/dotcms/inference/model/InferenceUsage.java b/dotCMS/src/main/java/com/dotcms/inference/model/InferenceUsage.java new file mode 100644 index 000000000000..87806a8ae6f3 --- /dev/null +++ b/dotCMS/src/main/java/com/dotcms/inference/model/InferenceUsage.java @@ -0,0 +1,26 @@ +package com.dotcms.inference.model; + +import java.io.Serializable; + +/** + * Tokens consumed by one exchange. + * + *Every field is nullable because not every provider reports usage. Absent counts are left + * absent and are never estimated — a fabricated number here would be indistinguishable from a + * real one to anyone reconciling spend.
+ * + * @param inputTokens tokens in the prompt, or null if unreported + * @param outputTokens tokens generated, or null if unreported + * @param totalTokens the sum as the provider reported it, or null + */ +public record InferenceUsage(Integer inputTokens, Integer outputTokens, Integer totalTokens) + implements Serializable { + + /** Usage the provider did not report. */ + public static final InferenceUsage UNREPORTED = new InferenceUsage(null, null, null); + + /** @return whether the provider reported any counts at all */ + public boolean isReported() { + return inputTokens != null || outputTokens != null || totalTokens != null; + } +} diff --git a/dotCMS/src/main/java/com/dotcms/inference/model/MultipleImagesUnsupportedException.java b/dotCMS/src/main/java/com/dotcms/inference/model/MultipleImagesUnsupportedException.java new file mode 100644 index 000000000000..6011e0fbf197 --- /dev/null +++ b/dotCMS/src/main/java/com/dotcms/inference/model/MultipleImagesUnsupportedException.java @@ -0,0 +1,27 @@ +package com.dotcms.inference.model; + +/** + * Raised when a caller asks for several images from a provider whose implementation only ever + * produces one. + * + *The capability is declared on the provider abstraction but not honoured by every + * implementation behind it, and the unsupported case announces itself by throwing from deep + * inside the client library. Left as it comes, that reaches the caller as an upstream failure — + * a retryable status for a request that cannot succeed no matter how many times it is sent. This + * type exists so the endpoint can tell the two apart and answer with a refusal naming + * {@code n} instead.
+ * + *This mirrors the adopted format's own behaviour, where {@code n} is accepted by the + * operation but rejected for models that cannot honour it.
+ */ +public class MultipleImagesUnsupportedException extends RuntimeException { + + private static final long serialVersionUID = 1L; + + /** + * @param modelName the model that cannot produce more than one image per request + */ + public MultipleImagesUnsupportedException(final String modelName) { + super("The image model '" + modelName + "' can only produce one image per request"); + } +} diff --git a/dotCMS/src/main/java/com/dotcms/inference/model/ResponseFormat.java b/dotCMS/src/main/java/com/dotcms/inference/model/ResponseFormat.java new file mode 100644 index 000000000000..c5f1286ef6b9 --- /dev/null +++ b/dotCMS/src/main/java/com/dotcms/inference/model/ResponseFormat.java @@ -0,0 +1,39 @@ +package com.dotcms.inference.model; + +import com.fasterxml.jackson.databind.JsonNode; + +import java.io.Serializable; + +/** + * The shape the caller wants the model's answer in. + * + * @param type free text, an arbitrary JSON object, or JSON conforming to a schema; required + * @param schema the JSON Schema the answer must satisfy; required when the type is + * {@link Type#JSON_SCHEMA} and null otherwise + */ +public record ResponseFormat(Type type, JsonNode schema) implements Serializable { + + /** The kind of output requested. */ + public enum Type { + /** Ordinary prose. */ + TEXT, + /** Any well-formed JSON object. */ + JSON_OBJECT, + /** JSON conforming to a supplied schema. */ + JSON_SCHEMA + } + + public ResponseFormat { + if (type == null) { + throw new IllegalArgumentException("ResponseFormat type is required"); + } + if (type == Type.JSON_SCHEMA && schema == null) { + throw new IllegalArgumentException("ResponseFormat type JSON_SCHEMA requires a schema"); + } + } + + /** @return the default, free-text format */ + public static ResponseFormat text() { + return new ResponseFormat(Type.TEXT, null); + } +} diff --git a/dotCMS/src/main/java/com/dotcms/inference/model/Role.java b/dotCMS/src/main/java/com/dotcms/inference/model/Role.java new file mode 100644 index 000000000000..0f1c3481375e --- /dev/null +++ b/dotCMS/src/main/java/com/dotcms/inference/model/Role.java @@ -0,0 +1,20 @@ +package com.dotcms.inference.model; + +/** + * The author of a single turn in an inference conversation. + * + *Deliberately free of any wire-format naming: the mapping between these constants and the + * strings a client sends lives in the REST layer, not here. See {@link InferenceRequest} for why + * the internal representation is kept independent of the format it is serialized to.
+ */ +public enum Role { + + /** Instruction supplied by the caller, ahead of the conversation. */ + SYSTEM, + /** A turn authored by the end user. */ + USER, + /** A turn authored by the model; may carry tool calls instead of content. */ + ASSISTANT, + /** The result of executing a tool the model asked for. */ + TOOL +} diff --git a/dotCMS/src/main/java/com/dotcms/inference/model/ToolChoice.java b/dotCMS/src/main/java/com/dotcms/inference/model/ToolChoice.java new file mode 100644 index 000000000000..fdffdb7dbb5d --- /dev/null +++ b/dotCMS/src/main/java/com/dotcms/inference/model/ToolChoice.java @@ -0,0 +1,49 @@ +package com.dotcms.inference.model; + +import java.io.Serializable; + +/** + * The caller's preference for whether, and which, tools the model should call. + * + * @param mode how freely the model may choose; required + * @param function the tool to force, required when and only when the mode is {@link Mode#FUNCTION} + */ +public record ToolChoice(Mode mode, String function) implements Serializable { + + /** How freely the model may pick a tool. */ + public enum Mode { + /** The model decides whether to call a tool. */ + AUTO, + /** The model must call some tool. */ + REQUIRED, + /** The model must not call a tool. */ + NONE, + /** The model must call the named tool. */ + FUNCTION + } + + public ToolChoice { + if (mode == null) { + throw new IllegalArgumentException("ToolChoice mode is required"); + } + if (mode == Mode.FUNCTION && (function == null || function.isBlank())) { + throw new IllegalArgumentException("ToolChoice mode FUNCTION requires a function name"); + } + if (mode != Mode.FUNCTION && function != null) { + throw new IllegalArgumentException("ToolChoice names a function only in FUNCTION mode"); + } + } + + /** @return a choice leaving the decision to the model */ + public static ToolChoice auto() { + return new ToolChoice(Mode.AUTO, null); + } + + /** + * @param function the tool the model must call + * @return a choice forcing one named tool + */ + public static ToolChoice function(final String function) { + return new ToolChoice(Mode.FUNCTION, function); + } +} diff --git a/dotCMS/src/main/java/com/dotcms/inference/rest/BearerOnlyAuthFilter.java b/dotCMS/src/main/java/com/dotcms/inference/rest/BearerOnlyAuthFilter.java new file mode 100644 index 000000000000..b2048297f385 --- /dev/null +++ b/dotCMS/src/main/java/com/dotcms/inference/rest/BearerOnlyAuthFilter.java @@ -0,0 +1,78 @@ +package com.dotcms.inference.rest; + +import com.dotcms.inference.model.InferenceError; +import com.dotcms.inference.rest.view.InferenceErrorView; + +import javax.ws.rs.container.ContainerRequestContext; +import javax.ws.rs.container.ContainerRequestFilter; +import javax.ws.rs.core.HttpHeaders; +import javax.ws.rs.core.MediaType; +import javax.ws.rs.core.Response; +import javax.ws.rs.ext.Provider; +import java.io.IOException; +import java.util.Optional; + +/** + * Refuses any credential on this family other than a bearer token. + * + *Without this, the surrounding dotCMS authentication accepts what it accepts everywhere else: + * {@code WebResource.authenticate} falls through to {@code PortalUtil.getUser(request)}, so a live + * session cookie — or basic auth — authenticates a caller who sent no {@code Authorization} header + * at all.
+ * + *That matters more here than it would elsewhere, because it is load-bearing for a decision + * made on the assumption that it was already true. This family emits no cross-origin headers, and + * the stated reason is that its credential is a long-lived token someone deliberately issued and + * placed on a server — not an ambient credential a browser attaches on its own. If a session + * cookie authenticates, that reasoning collapses: any page the user has open is one fetch away + * from spending the site's AI budget, and the absence of CORS headers becomes a formality rather + * than a control.
+ * + *The rule itself lives in {@link #bearerCredentialProblem(String)} rather than in this filter, + * and the resources call it directly as well. A filter alone would be a guarantee that exists only + * inside the JAX-RS chain — invisible to anything invoking a resource method directly, which is + * how this family's integration tests reach it. A guarantee that cannot be tested where it is + * relied upon is not much of a guarantee. The filter earns its place by applying automatically to + * every resource in the family, including ones not yet written.
+ */ +@Provider +@InferenceEndpoint +public class BearerOnlyAuthFilter implements ContainerRequestFilter { + + /** The only credential scheme this family accepts. */ + static final String BEARER_PREFIX = "Bearer "; + + @Override + public void filter(final ContainerRequestContext requestContext) throws IOException { + bearerCredentialProblem(requestContext.getHeaderString(HttpHeaders.AUTHORIZATION)) + .ifPresent(error -> requestContext.abortWith( + Response.status(error.httpStatus()) + .entity(InferenceErrorView.of(error)) + .type(MediaType.APPLICATION_JSON) + .build())); + } + + /** + * Checks a credential, without deciding what to do about a bad one. + * + *Returns the problem rather than throwing, so the filter can abort the exchange and a + * resource can fold it into whatever it already does with errors, from one implementation of + * the rule.
+ * + * @param authorizationHeader the request's Authorization header, or null when absent + * @return the refusal to send, or empty when the credential is an acceptable bearer token + */ + public static OptionalByte compatibility with the external standard is the whole point of this family, so nothing + * here is wrapped in the dotCMS {@code ResponseEntityView} envelope: a client library has to be + * able to deserialize both the answer and the refusal into its own types with no adapter. That is + * also why refusals are rendered as {@link InferenceErrorView} rather than left to dotCMS's + * generic exception mappers.
+ * + *This class is deliberately thin. Every decision that could be made differently by a second + * wire format lives elsewhere — the vocabulary translation in {@link ChatCompletionMapper}, the + * frame shapes in {@link SseSerializer}, the provider exchange in {@link InferenceAIClient}, and + * site plus configuration resolution in {@link AiHostResolver}. What is left here, and only here, + * is the HTTP contract: who may call, which site pays, which model is allowed, how many streams a + * node will hold open at once, and how a failure becomes a status.
+ * + *Two behaviours are worth calling out because getting either wrong is silent:
+ * + *The entity is a {@link StreamingOutput} rather than an already-written body: the frames + * have to reach the caller as the model produces them, which is the only thing that makes + * streaming worth its cost in held threads. That held thread is also why the concurrency + * ceiling is claimed here, before the response is returned — once the container starts writing + * the body, a refusal can no longer be expressed as a status.
+ * + * @param context the caller, the serving site and its configuration + * @param inferenceRequest the completion to run + * @return the event stream, or a 429 refusal when the node is already at its ceiling + */ + private Response streamCompletion(final ResolvedAiContext context, + final InferenceRequest inferenceRequest) { + + final int maxConcurrentStreams = InferenceLimits.current().maxConcurrentStreams(); + if (ACTIVE_STREAMS.incrementAndGet() > maxConcurrentStreams) { + ACTIVE_STREAMS.decrementAndGet(); + Logger.warn(this, "Refusing a streamed completion: the node is already serving its " + + "ceiling of " + maxConcurrentStreams + " concurrent streams"); + return errorResponse(new InferenceError( + "rate_limit_error", + "This node is already serving its ceiling of " + maxConcurrentStreams + + " concurrent streamed completions; retry shortly", + null, + 429)); + } + + // Fixed for the whole stream: a client correlates the chunks it stitches together by the + // completion id, so re-deriving any of these per event would break the correlation. + final String completionId = COMPLETION_ID_PREFIX + UUID.randomUUID(); + final long createdEpochSeconds = Instant.now().getEpochSecond(); + final String model = inferenceRequest.model(); + + final StreamingOutput streamingOutput = output -> { + final SseFrameWriter writer = + new SseFrameWriter(output, completionId, model, createdEpochSeconds); + try { + InferenceAIClient.get().stream(context.config(), inferenceRequest, writer); + writer.close(); + } finally { + // In a finally so a stream that failed, or a caller that walked away mid-answer, + // gives its slot back instead of leaking it for the life of the node. + ACTIVE_STREAMS.decrementAndGet(); + } + }; + + return Response.ok(streamingOutput).type(EVENT_STREAM).build(); + } + + /** + * Refuses a request whose model the resolved site has not configured. + * + *Applied to every caller, administrators included. The check is not about privilege — it + * is what stops a site's credentials being spent on a model its owner never chose, and an + * administrator of one site is not thereby entitled to spend another site's budget on an + * arbitrary model name.
+ * + * @param context the caller, the serving site and its configuration + * @param requestView the inbound payload + * @return a refusal, or null when the requested model is configured + */ + private Response refuseUnconfiguredModel(final ResolvedAiContext context, + final ChatCompletionRequestView requestView) { + + final String requestedModel = requestView == null ? null : requestView.model(); + if (StringUtils.isBlank(requestedModel)) { + // Named here rather than left to the mapper so the refusal can point at the field; the + // mapper's contract is a message, and a caller sent back to guessing is a support call. + return errorResponse(InferenceError.invalidRequest( + "The model field is required; there is no implicit default model", "model")); + } + + final ListA site with no usable chat configuration yields an empty list rather than an exception, + * which lands the caller on the same 404 as an unknown model. That is the honest answer: from + * where the caller stands, a model nobody configured and a model on a site nobody configured + * are the same absence, and distinguishing them would tell an unauthenticated-adjacent caller + * which sites have dotAI set up.
+ * + * @param context the caller, the serving site and its configuration + * @return the configured chat model names, in fallback order; empty when there are none + */ + private ListKept as a type rather than a lambda because ending the stream correctly needs two facts + * the writing itself produces: which event came last, and whether the connection is still + * usable. Both are written on the provider's threads and read on the request thread once + * {@link InferenceAIClient#stream} has returned, so both are volatile.
+ */ + private static final class SseFrameWriter implements ConsumerThe terminal marker follows a stream that ended normally and is withheld from one + * that did not — that withholding is the entire failure signal a streamed answer has left + * once its status line is gone. A stream that produced nothing at all gets an error frame + * of its own, for the same reason: silence and success look identical to a reader.
+ */ + private void close() { + final InferenceStreamEvent last = this.lastEvent; + + if (last == null) { + writeQuietly(SseSerializer.toFrame( + new InferenceStreamEvent.Error(InferenceError.upstream( + "The model provider produced no answer")), + completionId, + model, + createdEpochSeconds)); + return; + } + + if (SseSerializer.shouldWriteDoneMarker(last)) { + writeQuietly(SseSerializer.DONE_MARKER); + } + } + + /** + * @param frame the frame to write + * @throws UncheckedIOException when the caller has gone away, which ends the stream: the + * client is the only channel a streamed answer has, so there + * is nothing left to do but stop + */ + private void write(final String frame) { + if (broken) { + return; + } + try { + output.write(frame.getBytes(StandardCharsets.UTF_8)); + output.flush(); + } catch (final IOException e) { + broken = true; + throw new UncheckedIOException(e); + } + } + + /** + * Writes a closing frame, tolerating a connection that has already gone. + * + * @param frame the frame to write + */ + private void writeQuietly(final String frame) { + try { + write(frame); + } catch (final UncheckedIOException e) { + Logger.warn(ChatCompletionsResource.class, + "Could not close the inference stream; the caller has gone away"); + } + } + } +} diff --git a/dotCMS/src/main/java/com/dotcms/inference/rest/EmbeddingsResource.java b/dotCMS/src/main/java/com/dotcms/inference/rest/EmbeddingsResource.java new file mode 100644 index 000000000000..d478c1c3c4e8 --- /dev/null +++ b/dotCMS/src/main/java/com/dotcms/inference/rest/EmbeddingsResource.java @@ -0,0 +1,418 @@ +package com.dotcms.inference.rest; + +import com.dotcms.ai.client.langchain4j.InferenceAIClient; +import com.dotcms.ai.client.langchain4j.LangChain4jAIClient; +import com.dotcms.ai.rest.AiHostResolver; +import com.dotcms.ai.rest.ResolvedAiContext; +import com.dotcms.cost.RequestCost; +import com.dotcms.cost.RequestPrices.Price; +import com.dotcms.inference.model.InferenceError; +import com.dotcms.inference.model.InferenceUsage; +import com.dotcms.inference.rest.view.EmbeddingListView; +import com.dotcms.inference.rest.view.EmbeddingsRequestView; +import com.dotcms.inference.rest.view.InferenceErrorView; +import com.dotcms.rest.WebResource; +import com.dotcms.rest.annotation.NoCache; +import com.dotmarketing.exception.DotSecurityException; +import com.dotmarketing.util.Logger; +import com.fasterxml.jackson.databind.JsonNode; +import com.liferay.portal.model.User; +import io.swagger.v3.oas.annotations.Operation; +import io.swagger.v3.oas.annotations.media.Content; +import io.swagger.v3.oas.annotations.media.Schema; +import io.swagger.v3.oas.annotations.parameters.RequestBody; +import io.swagger.v3.oas.annotations.responses.ApiResponse; +import io.swagger.v3.oas.annotations.responses.ApiResponses; +import io.swagger.v3.oas.annotations.tags.Tag; +import org.apache.commons.lang3.StringUtils; +import org.glassfish.jersey.server.JSONP; + +import javax.servlet.http.HttpServletRequest; +import javax.servlet.http.HttpServletResponse; +import javax.ws.rs.Consumes; +import javax.ws.rs.POST; +import javax.ws.rs.Path; +import javax.ws.rs.Produces; +import javax.ws.rs.QueryParam; +import javax.ws.rs.core.Context; +import javax.ws.rs.core.HttpHeaders; +import javax.ws.rs.core.MediaType; +import javax.ws.rs.core.Response; +import java.util.ArrayList; +import java.util.List; +import java.util.Optional; + +/** + * Serves {@code POST /api/inference/v1/embeddings} — the embeddings endpoint of the + * OpenAI-wire-format family. + * + *Two things here are not a copy of the sibling completions endpoint with a different noun.
+ * + *The model gate reads the site's {@code embeddings} section, not its chat models. + * A site configures the two separately, so validating against chat would accept a chat model for an + * embeddings call and refuse the embeddings model the site actually configured — and would do so + * while returning a perfectly well-shaped 200 for the wrong request.
+ * + *{@code input} is a string or an array of strings. Batching is how content is + * ordinarily embedded — anyone indexing a site sends an array — and the array form is what makes + * the response a list of more than one entry, and therefore what makes {@code index} load-bearing: + * it is the caller's only means of correlating a vector back to the text they sent. That is also + * why a batch with one bad element is refused whole rather than cleaned: dropping an element would + * shift the index of every entry after it, and the caller would correlate against the wrong + * text.
+ * + *Blank and whitespace-only strings are refused wherever they appear. There is no meaningful + * embedding of nothing, and providers differ in whether they error on it or hand back a zero + * vector — which is exactly the inconsistency this family exists to hide, so it cannot be left to + * whichever provider a site configured.
+ * + *Nothing here logs the input text: it is customer content by definition, and it is not echoed + * into a refusal either — a message naming the offending position tells the caller what + * they need without repeating what they sent.
+ */ +@Path("/inference/v1/embeddings") +@Tag(name = "AI", description = "AI-powered content generation and analysis endpoints") +public class EmbeddingsResource { + + /** Section of the site's {@code providerConfig} JSON that configures embeddings. */ + private static final String EMBEDDINGS_SECTION = "embeddings"; + + /** The field a refusal about what to embed names. */ + private static final String INPUT_PARAM = "input"; + + /** + * Header form of the site override. Server-side callers routinely sit behind a proxy that + * rewrites the Host header, and a header is the only override they can set without rewriting + * the URL a standard client library builds. It wins over the query parameter because it is + * the more specific of the two. + */ + private static final String SITE_HEADER = "X-dotCMS-Site"; + + /** What a caller is told when the provider failed; never the provider's own words. */ + private static final String UPSTREAM_FAILURE_MESSAGE = + "The model provider failed to complete the request"; + + /** Longest model name echoed back in a refusal, so a huge value cannot be reflected whole. */ + private static final int MAX_ECHOED_MODEL_LENGTH = 120; + + /** + * Embeds one string, or a batch of them. + * + * @param request the inbound request + * @param response the outbound response, used only by the authentication handshake + * @param siteId optional site id or host name whose dotAI configuration should serve the + * request; the {@code X-dotCMS-Site} header overrides it + * @param requestView what to embed, in the standard wire shape + * @return the vectors, or an {@link InferenceErrorView} refusal + */ + @Operation( + operationId = "createEmbeddings", + summary = "Create embeddings", + description = "Embeds text against the model the resolved site has configured for " + + "embeddings, in the OpenAI-compatible request and response shape. The input " + + "field accepts either a single string or an array of strings embedded as one " + + "batch; the response is always a list with one entry per input, each carrying " + + "the index of the input it corresponds to. An absent, null or empty input is " + + "refused, as is a blank or whitespace-only string wherever it appears, and " + + "every element of an array must be a string — arrays of token ids are not " + + "supported. The model field is required and is validated against the site's " + + "embeddings configuration, not its chat models; there is no implicit default. " + + "Every response reports the serving site in the X-dotCMS-Resolved-Site header." + ) + @ApiResponses(value = { + @ApiResponse(responseCode = "200", + description = "The vectors, one per input", + content = @Content(mediaType = MediaType.APPLICATION_JSON, + schema = @Schema(implementation = EmbeddingListView.class))), + @ApiResponse(responseCode = "400", + description = "Malformed request, or one asking for something unsupported", + content = @Content(mediaType = MediaType.APPLICATION_JSON, + schema = @Schema(implementation = InferenceErrorView.class))), + @ApiResponse(responseCode = "401", + description = "Unauthorized - authentication required", + content = @Content(mediaType = MediaType.APPLICATION_JSON, + schema = @Schema(implementation = InferenceErrorView.class))), + @ApiResponse(responseCode = "403", + description = "Forbidden - the caller cannot read the requested site", + content = @Content(mediaType = MediaType.APPLICATION_JSON, + schema = @Schema(implementation = InferenceErrorView.class))), + @ApiResponse(responseCode = "404", + description = "The requested model is not configured for the resolved site", + content = @Content(mediaType = MediaType.APPLICATION_JSON, + schema = @Schema(implementation = InferenceErrorView.class))), + @ApiResponse(responseCode = "502", + description = "The model provider failed to complete the request", + content = @Content(mediaType = MediaType.APPLICATION_JSON, + schema = @Schema(implementation = InferenceErrorView.class))) + }) + @POST + @JSONP + @NoCache + @InferenceEndpoint + @RequestCost(Price.HTTP_FETCH) + @Consumes(MediaType.APPLICATION_JSON) + @Produces(MediaType.APPLICATION_JSON) + public Response embeddings(@Context final HttpServletRequest request, + @Context final HttpServletResponse response, + @QueryParam("siteId") final String siteId, + @RequestBody(description = "What to embed", + content = @Content(schema = @Schema( + implementation = EmbeddingsRequestView.class))) + final EmbeddingsRequestView requestView) { + + // Bearer only. Checked here as well as in BearerOnlyAuthFilter because the filter runs only + // inside the JAX-RS chain, while the rule has to hold wherever this method is reached. + final OptionalEach entry is stamped with the position of the input it embedded, which is why the batch + * is sent and received as one ordered whole rather than input by input.
+ * + * @param batch what the provider produced + * @param requestedModel the model the caller named, used when the provider reported none + * @return the answer + */ + private static EmbeddingListView toView(final InferenceAIClient.EmbeddingBatch batch, + final String requestedModel) { + final ListApplied to every caller, administrators included. The check is not about privilege — it is + * what stops a site's credentials being spent on a model its owner never chose for this + * operation. A site's chat model is refused here however well configured it is for chat.
+ * + * @param context the caller, the serving site and its configuration + * @param requestView the inbound payload + * @return a refusal, or null when the requested model is configured + */ + private Response refuseUnconfiguredModel(final ResolvedAiContext context, + final EmbeddingsRequestView requestView) { + + final String requestedModel = requestView == null ? null : requestView.model(); + if (StringUtils.isBlank(requestedModel)) { + return errorResponse(InferenceError.invalidRequest( + "The model field is required; there is no implicit default model", "model")); + } + + final ListEvery element of an array is inspected rather than only the first: a mixed array is the + * shape that actually arrives when a caller's collection was assembled from two sources, and + * an implementation that stopped at {@code input.get(0)} would send it upstream to fail as + * whatever the provider client makes of a heterogeneous list.
+ * + *When one element is at fault the message says which. {@code param} can only ever say + * {@code input} — that is the field the caller sent — so the message is the only place the + * position can appear, and a caller who batched five hundred strings should not have to bisect + * their own payload to find the bad one. The position is 0-based, the same number the + * response's own {@code index} correlates on.
+ * + * @param input the {@code input} value as it arrived, possibly null + * @return the refusal, or empty when there is something to embed + */ + private static OptionalThe image comes back inline, as base64, and never as a URL. A hosted URL + * would mean deciding storage, authentication and lifetime for an artifact generated from a prompt + * that may carry customer data, so this family declines to create a separately-addressable + * artifact at all. The answer shape has no {@code url} component, which is what keeps the decision + * from reversing quietly — a provider's own link passed through would otherwise be a 200 with a + * perfectly plausible body. Where a provider offers the choice dotCMS also asks it for the inline + * form, so nothing is minted upstream either; where a provider returns a link regardless, dotCMS + * fetches and re-encodes it rather than refuse the provider. On those providers an artifact really + * does exist upstream: the guarantee is that a caller never receives one.
+ * + *{@code n} is honoured, and omitting it means one. The adopted format + * supports generating several images in one request, so refusing outright would be this family + * declining something the standard supports. Three things bound it, and all three are refusals + * naming the field rather than quiet clamps — clamping answers a request for four hundred images + * with four, and a bill, without the caller ever learning that what they asked for is not what + * they got. A value below one is refused. A value above the configured ceiling is refused; the + * default ceiling is the one the standard itself documents for this field. And a request for + * several images against a model that can only ever produce one is refused here rather than + * translated as an upstream failure, because a retryable status would send a client's back-off + * into retrying something that cannot succeed. Support is asked of the model itself rather than + * read from a per-vendor table kept in this file, which would rot the first time a library + * release changed it.
+ * + *{@code size} is passed through to the provider. It changes both what the + * caller receives and what the site pays, so dropping it is a cost and correctness failure rather + * than the harmless compatibility courtesy that ignoring an incidental field is — and it is + * invisible in the response, since a provider asked for nothing in particular still returns a + * well-formed image.
+ * + *The model gate reads the site's {@code image} section, not its chat models + * and not its embeddings model. A site configures all three separately, so reusing another + * section's gate would accept the wrong model and refuse the right one while still answering + * 200 and 404 in the right shapes.
+ * + *Nothing here logs the prompt: it is customer content, and it is not echoed into a refusal + * either.
+ */ +@Path("/inference/v1/images") +@Tag(name = "AI", description = "AI-powered content generation and analysis endpoints") +public class ImagesResource { + + /** Section of the site's {@code providerConfig} JSON that configures image generation. */ + private static final String IMAGE_SECTION = "image"; + + /** How many images a request that says nothing about it asks for. */ + private static final int DEFAULT_IMAGE_COUNT = 1; + + /** The field a refusal about how many images to generate names. */ + private static final String COUNT_PARAM = "n"; + + /** The field a refusal about what to generate names. */ + private static final String PROMPT_PARAM = "prompt"; + + /** + * Header form of the site override. Server-side callers routinely sit behind a proxy that + * rewrites the Host header, and a header is the only override they can set without rewriting + * the URL a standard client library builds. It wins over the query parameter because it is + * the more specific of the two. + */ + private static final String SITE_HEADER = "X-dotCMS-Site"; + + /** What a caller is told when the provider failed; never the provider's own words. */ + private static final String UPSTREAM_FAILURE_MESSAGE = + "The model provider failed to complete the request"; + + /** Longest model name echoed back in a refusal, so a huge value cannot be reflected whole. */ + private static final int MAX_ECHOED_MODEL_LENGTH = 120; + + /** + * Generates images from a prompt. + * + * @param request the inbound request + * @param response the outbound response, used only by the authentication handshake + * @param siteId optional site id or host name whose dotAI configuration should serve the + * request; the {@code X-dotCMS-Site} header overrides it + * @param requestView what to generate, in the standard wire shape + * @return the images as base64, or an {@link InferenceErrorView} refusal + */ + @Operation( + operationId = "createImageGeneration", + summary = "Generate images", + description = "Generates images against the model the resolved site has configured " + + "for images, in the OpenAI-compatible request and response shape. Every image " + + "is returned inline as b64_json: no hosted, separately-addressable " + + "artifact is created, and no url is ever returned. Both model and prompt are " + + "required, and model is validated against the site's image configuration " + + "rather than its chat or embeddings models; there is no implicit default. The " + + "n field is honored — omitting it means one image. A value below 1, a value " + + "above the configured maximum, or any value above 1 on a site whose image " + + "model can only produce one, is refused with a 400 naming the field. The size field is passed through to the provider as a WIDTHxHEIGHT " + + "string; where the site carries an image size setting the caller's value wins " + + "and the site's is the default. Every response reports the serving site in " + + "the X-dotCMS-Resolved-Site header." + ) + @ApiResponses(value = { + @ApiResponse(responseCode = "200", + description = "The generated images, each inline as base64", + content = @Content(mediaType = MediaType.APPLICATION_JSON, + schema = @Schema(implementation = ImageGenerationView.class))), + @ApiResponse(responseCode = "400", + description = "Malformed request, or one asking for something unsupported", + content = @Content(mediaType = MediaType.APPLICATION_JSON, + schema = @Schema(implementation = InferenceErrorView.class))), + @ApiResponse(responseCode = "401", + description = "Unauthorized - authentication required", + content = @Content(mediaType = MediaType.APPLICATION_JSON, + schema = @Schema(implementation = InferenceErrorView.class))), + @ApiResponse(responseCode = "403", + description = "Forbidden - the caller cannot read the requested site", + content = @Content(mediaType = MediaType.APPLICATION_JSON, + schema = @Schema(implementation = InferenceErrorView.class))), + @ApiResponse(responseCode = "404", + description = "The requested model is not configured for the resolved site", + content = @Content(mediaType = MediaType.APPLICATION_JSON, + schema = @Schema(implementation = InferenceErrorView.class))), + @ApiResponse(responseCode = "502", + description = "The model provider failed to complete the request", + content = @Content(mediaType = MediaType.APPLICATION_JSON, + schema = @Schema(implementation = InferenceErrorView.class))) + }) + @POST + @JSONP + @NoCache + @InferenceEndpoint + @RequestCost(Price.HTTP_FETCH) + @Path("/generations") + @Consumes(MediaType.APPLICATION_JSON) + @Produces(MediaType.APPLICATION_JSON) + public Response generations(@Context final HttpServletRequest request, + @Context final HttpServletResponse response, + @QueryParam("siteId") final String siteId, + @RequestBody(description = "What to generate", + content = @Content(schema = @Schema( + implementation = ImageGenerationRequestView.class))) + final ImageGenerationRequestView requestView) { + + // Bearer only. Checked here as well as in BearerOnlyAuthFilter because the filter runs only + // inside the JAX-RS chain, while the rule has to hold wherever this method is reached. + final OptionalApplied to every caller, administrators included. The check is not about privilege — it is + * what stops a site's credentials being spent on a model its owner never chose for this + * operation. A site's chat and embeddings models are both refused here, however well + * configured they are elsewhere on the same site.
+ * + * @param context the caller, the serving site and its configuration + * @param requestView the inbound payload + * @return a refusal, or null when the requested model is configured + */ + private Response refuseUnconfiguredModel(final ResolvedAiContext context, + final ImageGenerationRequestView requestView) { + + final String requestedModel = requestView == null ? null : requestView.model(); + if (StringUtils.isBlank(requestedModel)) { + return errorResponse(InferenceError.invalidRequest( + "The model field is required; there is no implicit default model", "model")); + } + + final ListBinds the family's request and response filters to these endpoints alone. Without the + * binding a {@code @Provider} filter applies to every JAX-RS response in dotCMS, which is + * emphatically not wanted: the size ceiling and the resolved-site header are properties of this + * family, not of the product.
+ */ +@NameBinding +@Retention(RetentionPolicy.RUNTIME) +public @interface InferenceEndpoint { +} diff --git a/dotCMS/src/main/java/com/dotcms/inference/rest/InferenceRequestAttributes.java b/dotCMS/src/main/java/com/dotcms/inference/rest/InferenceRequestAttributes.java new file mode 100644 index 000000000000..319b5facc87b --- /dev/null +++ b/dotCMS/src/main/java/com/dotcms/inference/rest/InferenceRequestAttributes.java @@ -0,0 +1,18 @@ +package com.dotcms.inference.rest; + +/** + * Request-scoped attribute keys shared between the resources and the family's filters. + * + *The resolved site is published here rather than returned, so the response filter can report + * it even on a response the resource never produced — an exception mapped to a 4xx, or a stream + * that failed after it began. FR-020 asks for the serving site on every response, and a + * value only the happy path can set would not deliver that.
+ */ +public final class InferenceRequestAttributes { + + /** Identifier of the site whose configuration served the request. */ + public static final String RESOLVED_SITE_ID = "com.dotcms.inference.resolvedSiteId"; + + private InferenceRequestAttributes() { + } +} diff --git a/dotCMS/src/main/java/com/dotcms/inference/rest/ModelsResource.java b/dotCMS/src/main/java/com/dotcms/inference/rest/ModelsResource.java new file mode 100644 index 000000000000..e2f2f4a31207 --- /dev/null +++ b/dotCMS/src/main/java/com/dotcms/inference/rest/ModelsResource.java @@ -0,0 +1,224 @@ +package com.dotcms.inference.rest; + +import com.dotcms.ai.client.langchain4j.LangChain4jAIClient; +import com.dotcms.ai.rest.AiHostResolver; +import com.dotcms.ai.rest.ResolvedAiContext; +import com.dotcms.cost.RequestCost; +import com.dotcms.cost.RequestPrices.Price; +import com.dotcms.inference.model.InferenceError; +import com.dotcms.inference.rest.view.InferenceErrorView; +import com.dotcms.inference.rest.view.ModelListView; +import com.dotcms.rest.WebResource; +import com.dotcms.rest.annotation.NoCache; +import com.dotmarketing.exception.DotSecurityException; +import com.dotmarketing.util.Logger; +import com.liferay.portal.model.User; +import io.swagger.v3.oas.annotations.Operation; +import io.swagger.v3.oas.annotations.media.Content; +import io.swagger.v3.oas.annotations.media.Schema; +import io.swagger.v3.oas.annotations.responses.ApiResponse; +import io.swagger.v3.oas.annotations.responses.ApiResponses; +import io.swagger.v3.oas.annotations.tags.Tag; +import org.apache.commons.lang3.StringUtils; +import org.glassfish.jersey.server.JSONP; + +import javax.servlet.http.HttpServletRequest; +import javax.servlet.http.HttpServletResponse; +import javax.ws.rs.GET; +import javax.ws.rs.Path; +import javax.ws.rs.Produces; +import javax.ws.rs.QueryParam; +import javax.ws.rs.core.Context; +import javax.ws.rs.core.HttpHeaders; +import javax.ws.rs.core.MediaType; +import javax.ws.rs.core.Response; +import java.time.Instant; +import java.util.ArrayList; +import java.util.List; +import java.util.Optional; + +/** + * Serves {@code GET /api/inference/v1/models} — model discovery for the OpenAI-wire-format family. + * + *This endpoint is the discovery mechanism the rest of the family depends on. There is no + * implicit default model and no reserved alias, so the names listed here are exactly the names + * {@link ChatCompletionsResource} accepts — a caller who wants "whatever this site runs" reads the + * list and takes the first entry. Two consequences follow, and both are deliberate.
+ * + *A site with no dotAI configuration — and on an instance with none at the system level either + * — gets an empty list with a 200. Falling back to whichever site happens to be configured would + * hand a caller model names their own site will refuse, and would disclose that some other site + * has dotAI set up.
+ * + *Nothing here is wrapped in the dotCMS {@code ResponseEntityView} envelope, and refusals are + * rendered as {@link InferenceErrorView} rather than left to dotCMS's generic exception mappers, + * for the reason the whole family exists: a client library has to deserialize both the answer and + * the refusal into its own types with no adapter.
+ */ +@Path("/inference/v1/models") +@Tag(name = "AI", description = "AI-powered content generation and analysis endpoints") +public class ModelsResource { + + /** Section of the site's {@code providerConfig} JSON the listed models come from. */ + private static final String CHAT_SECTION = "chat"; + + /** + * Header form of the site override. Server-side callers routinely sit behind a proxy that + * rewrites the Host header, and a header is the only override they can set without rewriting + * the URL a standard client library builds. It wins over the query parameter because it is + * the more specific of the two. + */ + private static final String SITE_HEADER = "X-dotCMS-Site"; + + /** + * Lists the models the resolved site has configured for chat. + * + * @param request the inbound request + * @param response the outbound response, used only by the authentication handshake + * @param siteId optional site id or host name whose dotAI configuration should be read; the + * {@code X-dotCMS-Site} header overrides it + * @return the listing, or an {@link InferenceErrorView} refusal + */ + @Operation( + operationId = "listInferenceModels", + summary = "List the models this site has configured", + description = "Returns the models the resolved site has configured for chat, including " + + "every entry of a fallback chain, in configured order — the first is the " + + "site's primary model. The list is exactly the set of values the chat " + + "completions endpoint accepts as \"model\": there is no implicit default and " + + "no reserved alias, and nothing synthetic is added. No model provider is " + + "contacted. A site with no AI configuration, on an instance with none at the " + + "system level either, returns an empty data array rather than another site's " + + "models. Every response reports the serving site in the " + + "X-dotCMS-Resolved-Site header." + ) + @ApiResponses(value = { + @ApiResponse(responseCode = "200", + description = "The configured models, possibly none", + content = @Content(mediaType = MediaType.APPLICATION_JSON, + schema = @Schema(implementation = ModelListView.class))), + @ApiResponse(responseCode = "400", + description = "The requested site could not be resolved", + content = @Content(mediaType = MediaType.APPLICATION_JSON, + schema = @Schema(implementation = InferenceErrorView.class))), + @ApiResponse(responseCode = "401", + description = "Unauthorized - authentication required", + content = @Content(mediaType = MediaType.APPLICATION_JSON, + schema = @Schema(implementation = InferenceErrorView.class))), + @ApiResponse(responseCode = "403", + description = "Forbidden - the caller cannot read the requested site", + content = @Content(mediaType = MediaType.APPLICATION_JSON, + schema = @Schema(implementation = InferenceErrorView.class))) + }) + @GET + @JSONP + @NoCache + @InferenceEndpoint + @RequestCost(Price.HTTP_FETCH) + @Produces(MediaType.APPLICATION_JSON) + public Response models(@Context final HttpServletRequest request, + @Context final HttpServletResponse response, + @QueryParam("siteId") final String siteId) { + + // Bearer only. Checked here as well as in BearerOnlyAuthFilter because the filter runs only + // inside the JAX-RS chain, while the rule has to hold wherever this method is reached. + // The list names the site's configured models, which is exactly the reconnaissance an + // anonymous caller should not get for free. + final OptionalEvery entry reports the same creation time, read once here. dotCMS serves a + * configuration, not a catalogue: it does not know when a vendor published a model, and + * inventing a plausible per-model date would be indistinguishable from a real one to anyone + * who trusted it.
+ * + * @param modelNames the configured names, in fallback order + * @return the listing + */ + private static ModelListView toView(final ListRemoving the legacy 4096-character prompt cap was right — it was a character count standing + * in for a model's context window, which is the wrong control in the wrong place. But it left + * request size unbounded, and dotCMS parses and forwards a payload before any provider gets the + * chance to reject it, so the memory cost lands on this node and the token cost on the site's + * bill first.
+ * + *Rejecting here, with a typed error naming the limit, rather than leaving it to the servlet + * container: a container-level rejection is invisible to the caller's client library and varies + * per deployment.
+ */ +@Provider +@InferenceEndpoint +public class RequestSizeLimitFilter implements ContainerRequestFilter { + + @Override + public void filter(final ContainerRequestContext requestContext) throws IOException { + final int declaredLength = requestContext.getLength(); + if (declaredLength < 0) { + // Chunked or unknown length; the resource enforces the ceiling as it reads. + return; + } + final InferenceLimits limits = InferenceLimits.current(); + if (limits.exceedsMaxRequestBytes(declaredLength)) { + final InferenceError error = new InferenceError("invalid_request_error", + "Request body of " + declaredLength + " bytes exceeds the maximum of " + + limits.maxRequestBytes() + " bytes", null, + Response.Status.REQUEST_ENTITY_TOO_LARGE.getStatusCode()); + requestContext.abortWith( + Response.status(error.httpStatus()) + .entity(InferenceErrorView.of(error)) + .type(MediaType.APPLICATION_JSON) + .build()); + } + } +} diff --git a/dotCMS/src/main/java/com/dotcms/inference/rest/ResolvedSiteHeaderFilter.java b/dotCMS/src/main/java/com/dotcms/inference/rest/ResolvedSiteHeaderFilter.java new file mode 100644 index 000000000000..32d2b02ba1de --- /dev/null +++ b/dotCMS/src/main/java/com/dotcms/inference/rest/ResolvedSiteHeaderFilter.java @@ -0,0 +1,48 @@ +package com.dotcms.inference.rest; + +import javax.servlet.http.HttpServletRequest; +import javax.ws.rs.container.ContainerRequestContext; +import javax.ws.rs.container.ContainerResponseContext; +import javax.ws.rs.container.ContainerResponseFilter; +import javax.ws.rs.core.Context; +import javax.ws.rs.ext.Provider; +import java.io.IOException; + +/** + * Reports which site's configuration served the request, on every response. + * + *This is the answer to the one real risk in keeping standard site resolution: the fallbacks + * are convenient and correct, but they make it possible for a request to be served by a site it + * never named, and nobody could tell. A header answers that for every call — including the ones + * that resolved perfectly, which is more than an error on the unmatched case would have done, + * since it also surfaces a host that matched the wrong site through an alias + * collision.
+ * + *A header rather than a body field on purpose. The payloads of this family have to + * deserialize into a standard client library's own result types with no adapter, and an extra + * top-level field risks strict deserializers; a header is invisible to them, is identical for + * streamed and non-streamed responses, survives on errors where a body field could not, and is + * readable by the proxy or log pipeline that would actually do the spend reconciliation.
+ */ +@Provider +@InferenceEndpoint +public class ResolvedSiteHeaderFilter implements ContainerResponseFilter { + + /** Names the site whose dotAI configuration served the request. */ + public static final String RESOLVED_SITE_HEADER = "X-dotCMS-Resolved-Site"; + + @Context + private HttpServletRequest request; + + @Override + public void filter(final ContainerRequestContext requestContext, + final ContainerResponseContext responseContext) throws IOException { + if (request == null) { + return; + } + final Object siteId = request.getAttribute(InferenceRequestAttributes.RESOLVED_SITE_ID); + if (siteId instanceof String resolved && !resolved.isBlank()) { + responseContext.getHeaders().putSingle(RESOLVED_SITE_HEADER, resolved); + } + } +} diff --git a/dotCMS/src/main/java/com/dotcms/inference/rest/mapper/ChatCompletionMapper.java b/dotCMS/src/main/java/com/dotcms/inference/rest/mapper/ChatCompletionMapper.java new file mode 100644 index 000000000000..55ee4df8d637 --- /dev/null +++ b/dotCMS/src/main/java/com/dotcms/inference/rest/mapper/ChatCompletionMapper.java @@ -0,0 +1,397 @@ +package com.dotcms.inference.rest.mapper; + +import com.dotcms.inference.model.FinishReason; +import com.dotcms.inference.model.InferenceMessage; +import com.dotcms.inference.model.InferenceRequest; +import com.dotcms.inference.model.InferenceResponse; +import com.dotcms.inference.model.InferenceToolCall; +import com.dotcms.inference.model.InferenceToolSpec; +import com.dotcms.inference.model.InferenceUsage; +import com.dotcms.inference.model.ResponseFormat; +import com.dotcms.inference.model.Role; +import com.dotcms.inference.model.ToolChoice; +import com.dotcms.inference.rest.view.ChatCompletionRequestView; +import com.dotcms.inference.rest.view.ChatCompletionView; +import com.fasterxml.jackson.databind.JsonNode; + +import java.util.ArrayList; +import java.util.List; +import java.util.Locale; + +/** + * Translates between the OpenAI-compatible chat-completions wire shape and dotCMS's + * provider-neutral inference model. + * + *The wire shape and {@link InferenceRequest} are kept deliberately independent of each other, + * so this class is the single place where the two vocabularies meet. Everything that identifies a + * turn — a tool call's provider-assigned id, the unparsed JSON text of its arguments, the JSON + * Schema of a declared tool — is carried across verbatim: re-deriving any of it (from position, + * say) would bake one serialization into the model and break the moment a second wire format is + * written from the same internal request.
+ * + *Validation policy is asymmetric on purpose. A field this family has no opinion about is + * ignored, because rejecting a client's default payload over a field that changes nothing is + * pure friction. A field that changes what the caller gets or pays for — {@code n} above all — is + * rejected loudly instead, since a request quietly served as something other than what was asked + * is worse than one that failed. Every validation error names the offending field so the caller + * is not sent back to guessing.
+ * + *Stateless and thread-safe; it holds nothing.
+ */ +public final class ChatCompletionMapper { + + private static final String FUNCTION_TYPE = "function"; + + /** Static utility; never instantiated. */ + private ChatCompletionMapper() { + throw new AssertionError("ChatCompletionMapper is a static utility"); + } + + /** + * Binds an inbound chat-completions payload to the internal request the rest of the pipeline + * speaks. + * + *Wire-level validation happens here rather than deeper in, so a malformed request fails + * before any provider is chosen and before anybody is billed for a round trip. Correlation of + * tool results against the calls that requested them is deliberately not repeated: + * {@link InferenceRequest}'s compact constructor already enforces it, and duplicating the rule + * would mean two places to keep in step.
+ * + * @param view the bound wire payload + * @return the equivalent internal request + * @throws IllegalArgumentException when the payload is missing a required field or asks for + * something this family cannot serve, the message naming the + * offending field + */ + public static InferenceRequest toInferenceRequest(final ChatCompletionRequestView view) { + + if (view == null) { + throw new IllegalArgumentException("A chat completion request body is required"); + } + if (view.model() == null || view.model().isBlank()) { + throw new IllegalArgumentException( + "The model field is required; there is no implicit default model"); + } + if (view.messages() == null || view.messages().isEmpty()) { + throw new IllegalArgumentException( + "The messages field requires at least one message to infer from"); + } + if (view.n() != null && view.n() > 1) { + throw new IllegalArgumentException( + "The field n is not supported; exactly one choice is returned, so n was " + + "rejected rather than silently ignored"); + } + + return InferenceRequest.builder(view.model()) + .messages(toInferenceMessages(view.messages())) + .tools(toToolSpecs(view.tools())) + .toolChoice(toToolChoice(view.toolChoice())) + .responseFormat(toResponseFormat(view.responseFormat())) + .stream(Boolean.TRUE.equals(view.stream())) + .includeUsageInStream(view.streamOptions() != null + && Boolean.TRUE.equals(view.streamOptions().includeUsage())) + .temperature(view.temperature()) + .maxOutputTokens(view.maxTokens()) + .topP(view.topP()) + .stopSequences(view.stop()) + .build(); + } + + /** + * Renders a completed answer in the wire shape standard clients deserialize. + * + *Two details here are the point of the method. The model reported is the one that actually + * served the request, not the one asked for, which is the only signal a caller has that a + * site's fallback chain moved on. And usage the provider never reported is left absent rather + * than rendered as zeros — a fabricated count is indistinguishable from a real one to anyone + * reconciling spend.
+ * + * @param response the completed internal response + * @return the wire view of that response + * @throws IllegalArgumentException when there is no response to render + */ + public static ChatCompletionView toView(final InferenceResponse response) { + + if (response == null) { + throw new IllegalArgumentException("An inference response is required to render a view"); + } + + final InferenceMessage message = response.message(); + final ChatCompletionView.MessageView messageView = new ChatCompletionView.MessageView( + wireRole(message.role()), + message.content(), + toToolCallViews(message.toolCalls())); + + final ChatCompletionView.ChoiceView choice = new ChatCompletionView.ChoiceView( + 0, messageView, wireFinishReason(response.finishReason())); + + return new ChatCompletionView( + response.id(), + ChatCompletionView.OBJECT, + response.createdEpochSeconds(), + response.model(), + List.of(choice), + toUsageView(response.usage())); + } + + // --------------------------------------------------------------------- + // Inbound + // --------------------------------------------------------------------- + + /** + * Maps the conversation, preserving turn order. + * + * @param messageViews the wire turns, oldest first + * @return the internal turns, in the same order + */ + private static List{@link FinishReason#ERROR} has no equivalent in this format — a failed exchange is an + * error event, not a finish reason — so it renders as absent rather than as an invented + * reason a client would read as a successful stop.
+ * + * @param finishReason the internal reason + * @return the wire string, or null when the reason has no wire equivalent + */ + private static String wireFinishReason(final FinishReason finishReason) { + + switch (finishReason) { + case STOP: + return "stop"; + case LENGTH: + return "length"; + case TOOL_CALLS: + return "tool_calls"; + case CONTENT_FILTER: + return "content_filter"; + default: + return null; + } + } + + /** + * @param toolCalls the calls the model is asking for + * @return the wire tool calls, or null when there are none, so the field is omitted entirely + */ + private static ListByte compatibility with the external standard is the whole point of this endpoint family, so + * every frame is {@code data: } + one compact JSON object + a blank line. The blank line is not + * decoration: an SSE reader only dispatches an event once it sees it, so a frame without it leaves + * the client blocked holding a chunk it already received.
+ * + *The render is an exhaustive switch over the sealed {@link InferenceStreamEvent}. That is + * deliberate: a sixth variant added later must fail to compile here rather than silently fall + * through to a default branch and reach the wire as something a client cannot read.
+ * + *The JSON is built with Jackson rather than concatenated, because the shapes nest and the + * values — answer text and tool-call argument fragments — are attacker-influenced strings that + * must be escaped correctly.
+ */ +public final class SseSerializer { + + /** + * The terminal marker, exactly as it goes on the wire. Clients detect end-of-stream by matching + * these bytes, so it is a constant rather than something assembled per call. + */ + public static final String DONE_MARKER = "data: [DONE]\n\n"; + + /** The {@code object} discriminator every streamed chunk carries. */ + public static final String CHUNK_OBJECT = "chat.completion.chunk"; + + private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper(); + + private static final String DATA_PREFIX = "data: "; + private static final String FRAME_TERMINATOR = "\n\n"; + private static final String FUNCTION_TYPE = "function"; + + private SseSerializer() { + throw new AssertionError("SseSerializer is a utility class and is not instantiable"); + } + + /** + * Renders one stream event as a complete server-sent event frame. + * + *The completion id, model and creation time are repeated on every chunk of one completion + * and never change mid-stream: a client correlates the chunks it is stitching together by that + * id, so re-deriving them per event would break the correlation.
+ * + * @param event the event to render + * @param completionId the id shared by every chunk of this completion + * @param model the model that served the completion + * @param createdEpochSeconds when the completion was created, in epoch seconds + * @return the frame, {@code data: } prefix and blank-line terminator included + */ + public static String toFrame(final InferenceStreamEvent event, + final String completionId, + final String model, + final long createdEpochSeconds) { + + final ObjectNode payload = switch (event) { + + case InferenceStreamEvent.ContentDelta contentDelta -> + contentChunk(contentDelta, completionId, model, createdEpochSeconds); + + case InferenceStreamEvent.ToolCallDelta toolCallDelta -> + toolCallChunk(toolCallDelta, completionId, model, createdEpochSeconds); + + case InferenceStreamEvent.Finish finish -> + finishChunk(finish, completionId, model, createdEpochSeconds); + + case InferenceStreamEvent.Usage usage -> + usageChunk(usage, completionId, model, createdEpochSeconds); + + case InferenceStreamEvent.Error error -> errorPayload(error); + }; + + return DATA_PREFIX + payload.toString() + FRAME_TERMINATOR; + } + + /** + * Answers whether the terminal {@code [DONE]} marker may follow this event. + * + *It may, except after {@link InferenceStreamEvent.Error}. Once the first frame is written + * the HTTP status is already on the wire and can no longer carry a failure, so withholding the + * marker is the only thing that stops a client which does not parse the error frame from + * reading a truncated answer as a complete one. Every other variant answers {@code true}; the + * question is only ever asked of the last event a stream produced.
+ * + * @param event the last event the stream produced + * @return whether to close the stream with {@link #DONE_MARKER} + */ + public static boolean shouldWriteDoneMarker(final InferenceStreamEvent event) { + + return switch (event) { + case InferenceStreamEvent.Error ignored -> false; + case InferenceStreamEvent.ContentDelta ignored -> true; + case InferenceStreamEvent.ToolCallDelta ignored -> true; + case InferenceStreamEvent.Finish ignored -> true; + case InferenceStreamEvent.Usage ignored -> true; + }; + } + + /** + * Builds a chunk carrying one fragment of the answer's text. + * + * @param contentDelta the fragment + * @param completionId the id shared by every chunk of this completion + * @param model the model that served the completion + * @param createdEpochSeconds when the completion was created, in epoch seconds + * @return the chunk + */ + private static ObjectNode contentChunk(final InferenceStreamEvent.ContentDelta contentDelta, + final String completionId, + final String model, + final long createdEpochSeconds) { + + final ObjectNode chunk = envelope(completionId, model, createdEpochSeconds); + final ObjectNode choice = firstChoiceOf(chunk); + + choice.putObject("delta").put("content", contentDelta.text()); + choice.putNull("finish_reason"); + + return chunk; + } + + /** + * Builds a chunk carrying one fragment of one tool call. + * + *Only the first fragment of a call carries its {@code id}, its {@code type} and the tool's + * name; a continuation fragment emits neither key at all, not even as JSON + * null, because a null id reads to a client as a different call starting. {@code index} is the + * only thing tying a continuation fragment back to the call it belongs to.
+ * + * @param toolCallDelta the fragment + * @param completionId the id shared by every chunk of this completion + * @param model the model that served the completion + * @param createdEpochSeconds when the completion was created, in epoch seconds + * @return the chunk + */ + private static ObjectNode toolCallChunk(final InferenceStreamEvent.ToolCallDelta toolCallDelta, + final String completionId, + final String model, + final long createdEpochSeconds) { + + final ObjectNode chunk = envelope(completionId, model, createdEpochSeconds); + final ObjectNode choice = firstChoiceOf(chunk); + final ObjectNode toolCall = choice.putObject("delta").putArray("tool_calls").addObject(); + + final boolean firstFragment = toolCallDelta.id() != null || toolCallDelta.name() != null; + + toolCall.put("index", toolCallDelta.index()); + + if (toolCallDelta.id() != null) { + toolCall.put("id", toolCallDelta.id()); + } + if (firstFragment) { + toolCall.put("type", FUNCTION_TYPE); + } + + final ObjectNode function = toolCall.putObject(FUNCTION_TYPE); + + if (toolCallDelta.name() != null) { + function.put("name", toolCallDelta.name()); + } + function.put("arguments", toolCallDelta.partialArguments()); + + choice.putNull("finish_reason"); + + return chunk; + } + + /** + * Builds the chunk that announces the end of generation. + * + *Its delta is empty: the finish chunk says why the answer stopped, it does not add to the + * answer. That distinction is what separates a completed answer from a truncated one.
+ * + * @param finish why generation stopped + * @param completionId the id shared by every chunk of this completion + * @param model the model that served the completion + * @param createdEpochSeconds when the completion was created, in epoch seconds + * @return the chunk + */ + private static ObjectNode finishChunk(final InferenceStreamEvent.Finish finish, + final String completionId, + final String model, + final long createdEpochSeconds) { + + final ObjectNode chunk = envelope(completionId, model, createdEpochSeconds); + final ObjectNode choice = firstChoiceOf(chunk); + + choice.putObject("delta"); + choice.put("finish_reason", wireFinishReason(finish.reason())); + + return chunk; + } + + /** + * Builds the usage chunk. + * + *Its {@code choices} array is present and empty, which is the shape standard clients + * recognise as a usage-only chunk. Counts the provider did not report are left absent rather + * than zeroed — a fabricated count is indistinguishable from a real one to anyone reconciling + * spend.
+ * + * @param usageEvent the counts + * @param completionId the id shared by every chunk of this completion + * @param model the model that served the completion + * @param createdEpochSeconds when the completion was created, in epoch seconds + * @return the chunk + */ + private static ObjectNode usageChunk(final InferenceStreamEvent.Usage usageEvent, + final String completionId, + final String model, + final long createdEpochSeconds) { + + final ObjectNode chunk = envelope(completionId, model, createdEpochSeconds); + chunk.putArray("choices"); + + final InferenceUsage usage = usageEvent.usage(); + + if (!usage.isReported()) { + chunk.putNull("usage"); + return chunk; + } + + final ObjectNode rendered = chunk.putObject("usage"); + + if (usage.inputTokens() != null) { + rendered.put("prompt_tokens", usage.inputTokens().intValue()); + } + if (usage.outputTokens() != null) { + rendered.put("completion_tokens", usage.outputTokens().intValue()); + } + if (usage.totalTokens() != null) { + rendered.put("total_tokens", usage.totalTokens().intValue()); + } + + return chunk; + } + + /** + * Builds the payload of a failed stream's last frame. + * + *It is the standard top-level error object and nothing else — no chunk envelope, because + * this frame is not a chunk of an answer. The conversion is + * {@link InferenceErrorView#of(com.dotcms.inference.model.InferenceError)}, the one place that + * owns the error's wire shape, so the streamed error stays byte-identical to the non-streamed + * one. That conversion is also what keeps the HTTP status out of the body: retryability rides + * on the response's status code, and the standard error shape has no field for it.
+ * + * @param error the failure + * @return the error payload + */ + private static ObjectNode errorPayload(final InferenceStreamEvent.Error error) { + + return OBJECT_MAPPER.valueToTree(InferenceErrorView.of(error.error())); + } + + /** + * Starts a chunk with the envelope every chunk carries. + * + * @param completionId the id shared by every chunk of this completion + * @param model the model that served the completion + * @param createdEpochSeconds when the completion was created, in epoch seconds + * @return the chunk, with no choices yet + */ + private static ObjectNode envelope(final String completionId, + final String model, + final long createdEpochSeconds) { + + final ObjectNode chunk = OBJECT_MAPPER.createObjectNode(); + + chunk.put("id", completionId); + chunk.put("object", CHUNK_OBJECT); + chunk.put("created", createdEpochSeconds); + chunk.put("model", model); + + return chunk; + } + + /** + * Adds the single choice a content, tool-call or finish chunk carries. + * + * @param chunk the chunk being built + * @return the choice at index 0 + */ + private static ObjectNode firstChoiceOf(final ObjectNode chunk) { + + final ObjectNode choice = chunk.putArray("choices").addObject(); + choice.put("index", 0); + + return choice; + } + + /** + * Maps an internal finish reason to its wire string. + * + * @param reason why generation stopped + * @return the wire string a standard client expects + * @throws IllegalArgumentException for {@link FinishReason#ERROR}, which has no wire + * equivalent: a failed stream ends with an error frame, not + * with a finish reason meaning "this failed" + */ + private static String wireFinishReason(final FinishReason reason) { + + return switch (reason) { + case STOP -> "stop"; + case LENGTH -> "length"; + case TOOL_CALLS -> "tool_calls"; + case CONTENT_FILTER -> "content_filter"; + case ERROR -> throw new IllegalArgumentException( + "FinishReason.ERROR has no wire equivalent; a failed stream ends with an " + + "error frame"); + }; + } +} diff --git a/dotCMS/src/main/java/com/dotcms/inference/rest/view/ChatCompletionRequestView.java b/dotCMS/src/main/java/com/dotcms/inference/rest/view/ChatCompletionRequestView.java new file mode 100644 index 000000000000..ad6ac3619580 --- /dev/null +++ b/dotCMS/src/main/java/com/dotcms/inference/rest/view/ChatCompletionRequestView.java @@ -0,0 +1,91 @@ +package com.dotcms.inference.rest.view; + +import com.fasterxml.jackson.annotation.JsonIgnoreProperties; +import com.fasterxml.jackson.annotation.JsonProperty; +import com.fasterxml.jackson.databind.JsonNode; +import io.swagger.v3.oas.annotations.media.Schema; + +import java.util.List; + +/** + * The inbound chat-completion request, in the wire shape clients send. + * + *Unknown properties are ignored so a client's default payload does not fail on a field this + * family has no opinion about. That tolerance stops at fields which change what the caller gets + * or pays for — those are rejected by the mapper rather than silently dropped, because a request + * that quietly did something other than what was asked is worse than one that failed.
+ * + * @param model the model to use; required, no implicit default + * @param messages the conversation, oldest first + * @param tools tools the model may call + * @param toolChoice the caller's tool preference + * @param responseFormat the requested output shape + * @param stream whether to stream + * @param streamOptions streaming options, notably whether to include usage + * @param temperature sampling temperature, passed through + * @param maxTokens ceiling on generated tokens, passed through + * @param topP nucleus sampling, passed through + * @param stop stop sequences, passed through + * @param n number of choices; unsupported and rejected rather than ignored + */ +@JsonIgnoreProperties(ignoreUnknown = true) +@Schema(description = "Chat completion request in the OpenAI-compatible shape") +public record ChatCompletionRequestView( + @JsonProperty("model") @Schema(description = "Model id, as listed by GET /models", example = "gpt-4o") String model, + @JsonProperty("messages") @Schema(description = "The conversation, oldest first") ListNot wrapped in the dotCMS {@code ResponseEntityView} envelope. Byte compatibility with the + * external standard is the whole point of this endpoint family, and a wrapper would stop the + * payload deserializing into the result type a client library already has.
+ * + * @param id identity for this completion + * @param object always {@code chat.completion} + * @param created creation time, epoch seconds + * @param model the model that actually served the request, after any fallback + * @param choices the answers; always exactly one, as several are not supported + * @param usage tokens consumed, omitted when the provider did not report them + */ +@JsonInclude(JsonInclude.Include.NON_NULL) +@Schema(description = "Chat completion response in the OpenAI-compatible shape") +public record ChatCompletionView( + @JsonProperty("id") @Schema(description = "Completion id", example = "chatcmpl-8f3a") String id, + @JsonProperty("object") @Schema(description = "Object type", example = "chat.completion") String object, + @JsonProperty("created") @Schema(description = "Epoch seconds") long created, + @JsonProperty("model") @Schema(description = "Model that served the request", example = "gpt-4o") String model, + @JsonProperty("choices") @Schema(description = "The answers") ListAlways a list, even for a single string, because the request accepts a batch and a caller + * should not have to branch on which form they sent to read the answer.
+ * + *{@code index} on each entry is the position of the input it embedded, not the position of the + * entry in {@code data}. It is the caller's only means of correlating a vector back to the text + * they sent, which is why a batch with one bad element is refused whole rather than embedded + * partially: dropping an element would shift every index after it.
+ * + * @param object always {@code list} + * @param model the model that actually served, after any fallback hop + * @param data one entry per input, in the order the inputs were given + * @param usage tokens consumed by the whole batch, omitted when the provider reported none + */ +@JsonInclude(JsonInclude.Include.NON_NULL) +@Schema(description = "Embeddings response in the OpenAI-compatible shape") +public record EmbeddingListView( + @JsonProperty("object") @Schema(description = "Object type", example = "list") String object, + @JsonProperty("model") @Schema(description = "Model that served the request", + example = "text-embedding-3-small") String model, + @JsonProperty("data") @Schema(description = "One entry per input") ListThere is no completion count: an embeddings call generates nothing, so the standard shape + * carries only the prompt and the total.
+ * + * @param promptTokens tokens across every input in the batch + * @param totalTokens the same figure the provider reported as the total + */ + @JsonInclude(JsonInclude.Include.NON_NULL) + @Schema(description = "Tokens consumed") + public record UsageView( + @JsonProperty("prompt_tokens") @Schema(description = "Tokens across every input") Integer promptTokens, + @JsonProperty("total_tokens") @Schema(description = "Total tokens") Integer totalTokens) { + } +} diff --git a/dotCMS/src/main/java/com/dotcms/inference/rest/view/EmbeddingsRequestView.java b/dotCMS/src/main/java/com/dotcms/inference/rest/view/EmbeddingsRequestView.java new file mode 100644 index 000000000000..3febbdb50d4d --- /dev/null +++ b/dotCMS/src/main/java/com/dotcms/inference/rest/view/EmbeddingsRequestView.java @@ -0,0 +1,31 @@ +package com.dotcms.inference.rest.view; + +import com.fasterxml.jackson.annotation.JsonIgnoreProperties; +import com.fasterxml.jackson.annotation.JsonProperty; +import com.fasterxml.jackson.databind.JsonNode; +import io.swagger.v3.oas.annotations.media.Schema; + +/** + * The inbound embeddings request, in the wire shape clients send. + * + *{@code input} is held as a {@link JsonNode} rather than as a {@code String} or a
+ * {@code List
Unknown properties are ignored so a client's default payload does not fail on a field this + * family has no opinion about — {@code encoding_format} and {@code user}, typically.
+ * + * @param model the embeddings model to use; required, no implicit default + * @param input a single string, or an array of strings to embed as one batch + */ +@JsonIgnoreProperties(ignoreUnknown = true) +@Schema(description = "Embeddings request in the OpenAI-compatible shape") +public record EmbeddingsRequestView( + @JsonProperty("model") @Schema(description = "Model id, as configured for this site's embeddings", + example = "text-embedding-3-small") String model, + @JsonProperty("input") @Schema(description = "A string, or an array of strings to embed as one batch", + example = "The quick brown fox") JsonNode input) { +} diff --git a/dotCMS/src/main/java/com/dotcms/inference/rest/view/ImageGenerationRequestView.java b/dotCMS/src/main/java/com/dotcms/inference/rest/view/ImageGenerationRequestView.java new file mode 100644 index 000000000000..e090e3b0c92a --- /dev/null +++ b/dotCMS/src/main/java/com/dotcms/inference/rest/view/ImageGenerationRequestView.java @@ -0,0 +1,33 @@ +package com.dotcms.inference.rest.view; + +import com.fasterxml.jackson.annotation.JsonIgnoreProperties; +import com.fasterxml.jackson.annotation.JsonProperty; +import io.swagger.v3.oas.annotations.media.Schema; + +/** + * The inbound image-generation request, in the wire shape clients send. + * + *{@code n} is boxed rather than a primitive because "not sent" is a value a caller can express + * and this family has to answer: omitting it means one image, while an {@code n} of {@code 0} is a + * mistake that must be refused. A primitive would turn the first into the second silently.
+ * + *Unknown properties are ignored so a client's default payload does not fail on a field this + * family has no opinion about. Notably {@code response_format} is among them: the answer is always + * base64, so a caller asking for a URL is told nothing here — the response itself is the + * answer.
+ * + * @param model the image model to use; required, no implicit default + * @param prompt what to generate; required, there is nothing to generate without it + * @param n how many images; at least {@code 1}, and omitting it means {@code 1} + * @param size the image size, passed through to the provider where it accepts one + */ +@JsonIgnoreProperties(ignoreUnknown = true) +@Schema(description = "Image generation request in the OpenAI-compatible shape") +public record ImageGenerationRequestView( + @JsonProperty("model") @Schema(description = "Model id, as configured for this site's images", + example = "dall-e-3") String model, + @JsonProperty("prompt") @Schema(description = "What to generate", + example = "A cat in a hammock") String prompt, + @JsonProperty("n") @Schema(description = "Number of images; at least 1, defaults to 1", example = "1") Integer n, + @JsonProperty("size") @Schema(description = "Image size", example = "1024x1024") String size) { +} diff --git a/dotCMS/src/main/java/com/dotcms/inference/rest/view/ImageGenerationView.java b/dotCMS/src/main/java/com/dotcms/inference/rest/view/ImageGenerationView.java new file mode 100644 index 000000000000..5c109a137f29 --- /dev/null +++ b/dotCMS/src/main/java/com/dotcms/inference/rest/view/ImageGenerationView.java @@ -0,0 +1,45 @@ +package com.dotcms.inference.rest.view; + +import com.fasterxml.jackson.annotation.JsonInclude; +import com.fasterxml.jackson.annotation.JsonProperty; +import io.swagger.v3.oas.annotations.media.Schema; + +import java.util.List; + +/** + * A generated image, in the wire shape clients deserialize. + * + *Note what this type does not have: a {@code url} component. That is the decision rather than + * an omission. A hosted link would mean deciding storage, authentication and lifetime for an + * artifact generated from a prompt that may carry customer data, so this family declines to create + * a separately-addressable artifact at all and hands the bytes back inline instead. Encoding that + * in the type — rather than in a branch that happens never to be taken — is what stops a provider's + * own URL being passed through later by an implementation that no longer remembers why it must + * not.
+ * + *Should a URL form ever be offered, it has to be authenticated and time-limited, and it would + * be a new component here rather than a reinstated passthrough.
+ * + * @param created epoch seconds + * @param data the generated images, one per image the caller asked for + */ +@JsonInclude(JsonInclude.Include.NON_NULL) +@Schema(description = "Image generation response in the OpenAI-compatible shape") +public record ImageGenerationView( + @JsonProperty("created") @Schema(description = "Epoch seconds") long created, + @JsonProperty("data") @Schema(description = "The generated images") ListDeliberately not wrapped in the dotCMS {@code ResponseEntityView} envelope. Byte + * compatibility with the external standard is the whole point of this endpoint family, and a + * dotCMS wrapper would stop the payload deserializing into the error types a client library + * already has.
+ * + *Note what is absent: there is no retryable field. The standard error shape has none, and + * inventing one would break that same compatibility — retryability is carried by the HTTP status, + * which is what a standard client's back-off actually keys off.
+ */ +@Schema(description = "Error response in the OpenAI-compatible error shape") +public record InferenceErrorView(@JsonProperty("error") @Schema(description = "The error") Body error) { + + /** + * The error itself. + * + * @param message a safe description; never the upstream provider's raw envelope + * @param type the error family, e.g. {@code invalid_request_error} + * @param param the offending request field, when one can be named + * @param code a machine-readable code; null unless the provider supplied one + */ + @JsonInclude(JsonInclude.Include.NON_NULL) + @Schema(description = "Error detail") + public record Body( + @JsonProperty("message") @Schema(description = "Human-readable description of the failure", + example = "The model 'gpt-4o' is not configured for this site") String message, + @JsonProperty("type") @Schema(description = "Error family", + example = "invalid_request_error") String type, + @JsonProperty("param") @Schema(description = "Offending request field, when applicable", + example = "model") String param, + @JsonProperty("code") @Schema(description = "Machine-readable code, when the provider supplies one") + String code) { + } + + /** + * @param error the internal error + * @return the same error in the wire shape, without its HTTP status, which the response carries + */ + public static InferenceErrorView of(final InferenceError error) { + return new InferenceErrorView( + new Body(error.message(), error.type(), error.param(), null)); + } +} diff --git a/dotCMS/src/main/java/com/dotcms/inference/rest/view/ModelListView.java b/dotCMS/src/main/java/com/dotcms/inference/rest/view/ModelListView.java new file mode 100644 index 000000000000..72edd00a2259 --- /dev/null +++ b/dotCMS/src/main/java/com/dotcms/inference/rest/view/ModelListView.java @@ -0,0 +1,55 @@ +package com.dotcms.inference.rest.view; + +import com.fasterxml.jackson.annotation.JsonProperty; +import io.swagger.v3.oas.annotations.media.Schema; + +import java.util.List; + +/** + * The models a site has configured, in the wire shape clients already parse. + * + *Not wrapped in the dotCMS {@code ResponseEntityView} envelope, for the same reason nothing + * else in this family is: a client library has to deserialize this into its own model-list type + * with no adapter.
+ * + *This listing is load-bearing rather than decorative. There is no implicit default model and + * no reserved alias, so the set of names here is exactly the set of {@code model} values the chat + * endpoint accepts — which makes it the only way a caller can learn what to ask for. Adding + * anything synthetic would advertise a name that endpoint refuses.
+ * + * @param object always {@code list} + * @param data one entry per configured model, in configured order; the first is the primary + */ +@Schema(description = "Model listing in the OpenAI-compatible shape") +public record ModelListView( + @JsonProperty("object") @Schema(description = "Object type", example = "list") String object, + @JsonProperty("data") @Schema(description = "The configured models, primary first") + ListThe cases here are the ones where a wire format and an internal model can silently diverge: + * role naming, tool-call identity, the unparsed JSON text of tool arguments, the JSON Schema of a + * declared tool, and the sampling parameters a caller pays for. Dropping any of those compiles + * fine and fails only against a real provider, which is why each gets its own assertion here + * rather than a round-trip smoke test.
+ * + *Most cases deserialize a realistic payload with Jackson rather than calling the view's + * canonical constructor, so the {@code @JsonProperty} names on the wire record — {@code max_tokens}, + * {@code tool_call_id}, {@code stream_options} — are exercised alongside the mapping itself.
+ */ +public class ChatCompletionRequestMapperTest { + + private static final ObjectMapper MAPPER = new ObjectMapper(); + + /** + * Deserializes a chat-completions payload into the inbound view. + * + * @param json the request body as a client would send it + * @return the bound view + * @throws Exception when the payload is not readable + */ + private static ChatCompletionRequestView parse(final String json) throws Exception { + return MAPPER.readValue(json, ChatCompletionRequestView.class); + } + + /** + * Given a conversation carrying every role the format defines, in order, + * When mapped to the internal request, + * Then each wire role becomes its {@link Role} constant and the turn order is preserved. + */ + @Test + public void test_toInferenceRequest_allFourRoles_mapsRolesAndPreservesOrder() throws Exception { + final String json = "{" + + "\"model\":\"gpt-4o\"," + + "\"messages\":[" + + "{\"role\":\"system\",\"content\":\"You are a helpful assistant.\"}," + + "{\"role\":\"user\",\"content\":\"What is the weather in Bogota?\"}," + + "{\"role\":\"assistant\",\"content\":null,\"tool_calls\":[" + + "{\"id\":\"call_1\",\"type\":\"function\",\"function\":" + + "{\"name\":\"get_weather\",\"arguments\":\"{\\\"city\\\":\\\"Bogota\\\"}\"}}]}," + + "{\"role\":\"tool\",\"tool_call_id\":\"call_1\",\"content\":\"{\\\"tempC\\\":19}\"}" + + "]}"; + + final InferenceRequest request = ChatCompletionMapper.toInferenceRequest(parse(json)); + + final ListCoverage:
+ *The first group exercises the internal representation directly — the compact constructors of + * {@link InferenceRequest} and {@link InferenceMessage}, which already enforce their invariants. + * These tests pass today and are here to pin that behaviour down so a later refactor of the model + * cannot quietly relax it.
+ * + *The second group exercises + * {@link ChatCompletionMapper#toInferenceRequest(ChatCompletionRequestView)}, which translates the + * wire shape into the internal one and is where wire-level validation belongs. That method is a + * skeleton at the time of writing, so these tests fail — deliberately. They are the specification + * the implementation is written against, in particular the FR-013 line: a field this family has no + * opinion about is ignored, but a field that changes what the caller gets or pays for — {@code n} + * being the example — must fail loudly rather than be silently dropped.
+ * + *Where a validation error is asserted, the assertion is on the error naming the offending + * field or identity. A validation error that does not say what was wrong sends the caller back to + * guessing, which is most of what these rules exist to prevent.
+ * + * @see InferenceRequest + * @see InferenceMessage + * @see ChatCompletionMapper + */ +public class InferenceRequestValidationTest { + + private static final String VALID_MODEL = "gpt-4o"; + + // --------------------------------------------------------------------- + // Internal representation — InferenceRequest + // --------------------------------------------------------------------- + + /** + * Given a request built with a blank model, + * When the record is constructed, + * Then it is rejected naming {@code model} — there is no implicit default (FR-024). + */ + @Test + public void test_inferenceRequest_blankModel_throwsNamingModel() { + final ListMatched on a word boundary rather than a bare substring, so a message that happens to + * contain the letters of a short field name — {@code n} above all — does not pass for one that + * names the field.
+ * + * @param message the exception message under test + * @param token the field name or identity the message must name + */ + private static void assertMessageNamesToken(final String message, final String token) { + assertNotNull("Validation error carried no message at all", message); + final Pattern pattern = + Pattern.compile("(?Weld cannot intercept a final method: it proxies by subclassing, and a final method cannot be + * overridden. Annotating one with {@code @RequestCost} — an interceptor binding — makes Weld throw + * {@code WELD-001504} while building the injection target, which fails + * {@code DotRestApplication}'s servlet init. That does not break one endpoint; it takes down + * every REST endpoint in dotCMS, and the log then fills with a secondary + * "resource configuration is not modifiable" error from each retry, which looks like an entirely + * different problem and is where an investigation naturally starts. + * + *This exists because nothing else here would catch it. Every integration test in this family + * invokes the resource methods directly, so they never pass through Weld or Jersey at all — the + * whole suite was green while the application could not start. That blind spot is structural, not + * an oversight in any one test, so the guard has to be explicit.
+ * + *Asserted by reflection over every method rather than against a list of known ones: a fifth + * operation added later with {@code public final} is precisely the mistake this catches, and a + * test naming today's four would not.
+ */ +public class InterceptedMethodsAreNotFinalTest { + + /** + * Given every resource in this endpoint family + * When a method carries a CDI interceptor binding + * Then it is not final, so Weld can proxy it + */ + @Test + public void test_noInterceptedResourceMethodIsFinal() { + final ListOnce the first chunk is written the HTTP status is already on the wire and can no longer + * carry the failure. The error event is the only way a caller learns the answer is incomplete, + * and withholding the terminal {@code [DONE]} marker is what stops a client that + * does not parse the error event from reading a truncated answer as a finished one. That single + * behaviour — {@link SseSerializer#shouldWriteDoneMarker(InferenceStreamEvent)} returning + * {@code false} after an {@link InferenceStreamEvent.Error} — is the most important assertion in + * the streaming work.
+ * + *Coverage:
+ *This is the single most important assertion in the streaming work: withholding + * {@code [DONE]} is the only thing that stops a client which does not parse the error event + * from treating a truncated answer as a complete one. (FR-039)
+ */ + @Test + public void test_shouldWriteDoneMarker_errorEvent_returnsFalse() { + + final InferenceStreamEvent event = + new InferenceStreamEvent.Error(InferenceError.upstream(UPSTREAM_MESSAGE)); + + assertFalse("A failed stream must NOT be terminated with [DONE] — a client that does not " + + "parse the error event must see a truncated stream, never a cleanly " + + "finished one", + SseSerializer.shouldWriteDoneMarker(event)); + } + + /** + * Given the completion timeout expiring mid-generation, the second way a stream fails after it + * has started. + * When the serializer is asked whether the terminal marker follows. + * Then it says no, identically to the provider-failure path — the caller must not be able to + * tell a timed-out answer from a complete one only by luck. (FR-039) + */ + @Test + public void test_shouldWriteDoneMarker_completionTimeoutErrorEvent_returnsFalse() { + + final InferenceStreamEvent event = new InferenceStreamEvent.Error(completionTimeout()); + + assertFalse("A stream cut short by the completion timeout must NOT be terminated with " + + "[DONE] either", + SseSerializer.shouldWriteDoneMarker(event)); + } + + /** + * Given streams that ended normally — one on a finish event, one on a usage event. + * When the serializer is asked whether the terminal marker follows. + * Then it says yes for every serializable finish reason and for usage, so the {@code false} + * returned after an error is demonstrably specific to failure rather than blanket behaviour. + */ + @Test + public void test_shouldWriteDoneMarker_finishAndUsageEvents_returnTrue() { + + for (final FinishReason reason : FinishReason.values()) { + if (reason == FinishReason.ERROR) { + // Never serialized as a finish reason; a failed stream ends with an Error event. + continue; + } + final InferenceStreamEvent finish = new InferenceStreamEvent.Finish(reason); + assertTrue("A stream that finished with " + reason + " must be terminated with [DONE]", + SseSerializer.shouldWriteDoneMarker(finish)); + } + + final InferenceStreamEvent usage = + new InferenceStreamEvent.Usage(new InferenceUsage(82, 17, 99)); + assertTrue("A stream that ends with usage must still be terminated with [DONE]", + SseSerializer.shouldWriteDoneMarker(usage)); + + final InferenceStreamEvent error = + new InferenceStreamEvent.Error(InferenceError.upstream(UPSTREAM_MESSAGE)); + assertFalse("Only failure withholds the terminal marker", + SseSerializer.shouldWriteDoneMarker(error)); + } + + /** + * Given an error event. + * When it is rendered to a frame. + * Then the frame is neither equal to nor contains {@link SseSerializer#DONE_MARKER}, so a + * client scanning the byte stream for the terminator never finds one on a failed stream. + * (FR-039) + */ + @Test + public void test_toFrame_errorEvent_frameNeitherIsNorContainsDoneMarker() { + + final InferenceStreamEvent upstreamFailure = + new InferenceStreamEvent.Error(InferenceError.upstream(UPSTREAM_MESSAGE)); + final InferenceStreamEvent timeoutFailure = + new InferenceStreamEvent.Error(completionTimeout()); + + for (final InferenceStreamEvent event + : new InferenceStreamEvent[] {upstreamFailure, timeoutFailure}) { + + final String frame = SseSerializer.toFrame(event, COMPLETION_ID, MODEL, CREATED); + + assertNotNull("The serializer must render an error frame", frame); + assertTrue("An SSE frame is written behind the data: prefix", + frame.startsWith(DATA_PREFIX)); + assertNotEquals("An error frame is not the terminal marker", + SseSerializer.DONE_MARKER, frame); + assertFalse("An error frame must not carry the terminal marker", + frame.contains(SseSerializer.DONE_MARKER)); + assertFalse("An error frame must not carry the terminal marker's payload", + frame.contains("[DONE]")); + } + } + + /** + * Given the completion timeout expiring mid-generation. + * When the error event is rendered. + * Then it renders the same standard error shape as the provider-failure path — a top-level + * {@code error} object with {@code message} and {@code type}, and no status in the body — so + * the second way to fail mid-stream is indistinguishable in handling from the first. (FR-039) + * + * @throws Exception when the rendered frame cannot be parsed + */ + @Test + public void test_toFrame_completionTimeoutErrorEvent_rendersSameShapeAsUpstreamFailure() + throws Exception { + + final InferenceError timeout = completionTimeout(); + + final JsonNode frame = parseFrame(SseSerializer.toFrame( + new InferenceStreamEvent.Error(timeout), COMPLETION_ID, MODEL, CREATED)); + + final JsonNode error = frame.get("error"); + assertNotNull("A timed-out stream's last frame carries a top-level error object", error); + assertEquals("The timeout's safe description must be carried verbatim", + TIMEOUT_MESSAGE, error.path("message").asText()); + assertEquals("A timeout is reported in the same error family as any upstream failure", + timeout.type(), error.path("type").asText()); + assertFalse("The status must not leak into the body", error.has("httpStatus")); + assertFalse("The status must not leak into the body", error.has("status")); + } + + /** + * A timeout-flavoured error — the second way a stream fails once it has begun, when + * {@code DOT_INFERENCE_COMPLETION_TIMEOUT_SECONDS} expires mid-generation. + * + * @return the error the streaming path raises on a completion timeout + */ + private static InferenceError completionTimeout() { + return new InferenceError("api_error", TIMEOUT_MESSAGE, null, 504); + } + + /** + * Parses a rendered frame's JSON payload, stripping the {@code data: } prefix and the trailing + * newlines that terminate a server-sent event. + * + * @param frame the frame the serializer rendered + * @return the parsed payload + * @throws Exception when the payload is not parseable JSON + */ + private static JsonNode parseFrame(final String frame) throws Exception { + + assertNotNull("The serializer must render a frame", frame); + assertTrue("An SSE frame is written behind the data: prefix", frame.startsWith(DATA_PREFIX)); + + final String payload = frame.substring(DATA_PREFIX.length()).trim(); + return OBJECT_MAPPER.readTree(payload); + } +} diff --git a/dotCMS/src/test/java/com/dotcms/inference/SseSerializerTest.java b/dotCMS/src/test/java/com/dotcms/inference/SseSerializerTest.java new file mode 100644 index 000000000000..1fe73e172aed --- /dev/null +++ b/dotCMS/src/test/java/com/dotcms/inference/SseSerializerTest.java @@ -0,0 +1,256 @@ +package com.dotcms.inference; + +import com.dotcms.inference.model.FinishReason; +import com.dotcms.inference.model.InferenceStreamEvent; +import com.dotcms.inference.model.InferenceUsage; +import com.dotcms.inference.rest.mapper.SseSerializer; +import com.fasterxml.jackson.databind.JsonNode; +import com.fasterxml.jackson.databind.ObjectMapper; +import org.junit.Test; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; + +/** + * Unit tests for {@link SseSerializer}, which renders internal {@link InferenceStreamEvent}s as the + * server-sent events a standard chat-completions client reads from + * {@code /api/inference/v1/chat/completions} when {@code "stream": true}. + * + *This class covers the chunk envelope (FR-007); incremental tool calls (FR-008) are covered by + * {@code SseToolCallStreamTest}.
+ * + *Coverage:
+ *A tool call does not arrive whole. Its identity and name come on the first fragment, its + * arguments dribble in across later ones, and the only thing tying a continuation fragment to the + * call it belongs to is {@code index}. A client accumulates those fragments itself, so the + * serializer's job is to emit exactly what belongs to each fragment and nothing more.
+ * + *Coverage:
+ *In the chat-completions wire format the usage event is a chunk whose {@code choices} array is + * empty. Some stream readers assume every chunk has a non-empty {@code choices} + * and break on one that does not, which is why the event is emitted only when the caller asked + * for it via the standard streaming option — sending it unasked would break exactly the clients + * this endpoint family exists to support.
+ * + *Coverage:
+ *Streaming is where the wire format is easiest to get subtly wrong, because a client consumes + * the answer as it arrives and therefore cannot re-read it once it has acted on it. Three of the + * four behaviours here are about the frames themselves; the fourth is about how a stream + * ends, which is the only part a client cannot recover from on its own.
+ * + *The provider is a WireMock server standing in for an OpenAI-compatible endpoint, wired in + * through the same dotAI app secrets the rest of the AI integration tests use, so the exchange + * travels the real client path rather than a stubbed one.
+ * + *The usage tests pin down that {@code stream_options} is not forwarded + * upstream. It is an OpenAI-format option that four of dotCMS's seven providers do not + * understand, so forwarding it would make streamed usage work on some vendors and silently fail + * on others. dotCMS reads the flag itself and builds the chunk from the token counts the unified + * provider abstraction returns on completion — available whichever vendor served the request — + * and suppresses a usage chunk a provider volunteers unasked, since its empty {@code choices} + * array is the shape that breaks readers assuming every chunk carries one.
+ */ +public class ChatCompletionsStreamingTest { + + /** The chat model the site is configured with, and the only one these tests ask for. */ + private static final String CHAT_MODEL = "gpt-4o-mini"; + + /** Path an OpenAI-compatible provider serves completions on. */ + private static final String COMPLETIONS_PATH = "/chat/completions"; + + /** Media type a streamed completion is served as. */ + private static final String EVENT_STREAM = "text/event-stream"; + + /** Prefix every server-sent event field this family emits carries. */ + private static final String DATA_PREFIX = "data: "; + + /** Payload of the terminal frame, without its prefix or terminator. */ + private static final String DONE_PAYLOAD = "[DONE]"; + + /** The tool the streamed tool-call conversation declares. */ + private static final String TOOL_NAME = "get_weather"; + + /** The id the stubbed provider mints for its tool call, and that must survive verbatim. */ + private static final String TOOL_CALL_ID = "call_dot_weather_1"; + + /** The argument JSON the provider sends in fragments, once reassembled. */ + private static final String EXPECTED_TOOL_ARGUMENTS = "{\"city\":\"Bogota\"}"; + + /** The streaming option gating the usage chunk, as it appears on the wire. */ + private static final String INCLUDE_USAGE = "include_usage"; + + /** A plain content stream: a role chunk, two content chunks, a finish chunk, the marker. */ + private static final String PROVIDER_CONTENT_STREAM = """ + data: {"id":"chatcmpl-stream-1","object":"chat.completion.chunk","created":1789000000,"model":"gpt-4o-mini","choices":[{"index":0,"delta":{"role":"assistant","content":""}}]} + + data: {"id":"chatcmpl-stream-1","object":"chat.completion.chunk","created":1789000000,"model":"gpt-4o-mini","choices":[{"index":0,"delta":{"content":"It is currently "}}]} + + data: {"id":"chatcmpl-stream-1","object":"chat.completion.chunk","created":1789000000,"model":"gpt-4o-mini","choices":[{"index":0,"delta":{"content":"19 degrees in Bogota."}}]} + + data: {"id":"chatcmpl-stream-1","object":"chat.completion.chunk","created":1789000000,"model":"gpt-4o-mini","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]} + + data: [DONE] + + """; + + /** The same stream, with the usage chunk the provider adds when asked for it. */ + private static final String PROVIDER_CONTENT_STREAM_WITH_USAGE = """ + data: {"id":"chatcmpl-stream-1","object":"chat.completion.chunk","created":1789000000,"model":"gpt-4o-mini","choices":[{"index":0,"delta":{"role":"assistant","content":""}}]} + + data: {"id":"chatcmpl-stream-1","object":"chat.completion.chunk","created":1789000000,"model":"gpt-4o-mini","choices":[{"index":0,"delta":{"content":"It is currently 19 degrees in Bogota."}}]} + + data: {"id":"chatcmpl-stream-1","object":"chat.completion.chunk","created":1789000000,"model":"gpt-4o-mini","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]} + + data: {"id":"chatcmpl-stream-1","object":"chat.completion.chunk","created":1789000000,"model":"gpt-4o-mini","choices":[],"usage":{"prompt_tokens":82,"completion_tokens":17,"total_tokens":99}} + + data: [DONE] + + """; + + /** A tool call whose arguments are split across two chunks, as providers really send them. */ + private static final String PROVIDER_TOOL_CALL_STREAM = """ + data: {"id":"chatcmpl-stream-2","object":"chat.completion.chunk","created":1789000000,"model":"gpt-4o-mini","choices":[{"index":0,"delta":{"role":"assistant","content":null}}]} + + data: {"id":"chatcmpl-stream-2","object":"chat.completion.chunk","created":1789000000,"model":"gpt-4o-mini","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"id":"call_dot_weather_1","type":"function","function":{"name":"get_weather","arguments":"{\\"ci"}}]}}]} + + data: {"id":"chatcmpl-stream-2","object":"chat.completion.chunk","created":1789000000,"model":"gpt-4o-mini","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"ty\\":\\"Bogota\\"}"}}]}}]} + + data: {"id":"chatcmpl-stream-2","object":"chat.completion.chunk","created":1789000000,"model":"gpt-4o-mini","choices":[{"index":0,"delta":{},"finish_reason":"tool_calls"}]} + + data: [DONE] + + """; + + /** + * A stream that starts well and then breaks: the third frame is truncated JSON and the + * provider never sends a terminal marker. The generation is already in flight by then, which + * is exactly the window FR-039 is about. + */ + private static final String PROVIDER_MALFORMED_STREAM = """ + data: {"id":"chatcmpl-stream-3","object":"chat.completion.chunk","created":1789000000,"model":"gpt-4o-mini","choices":[{"index":0,"delta":{"role":"assistant","content":""}}]} + + data: {"id":"chatcmpl-stream-3","object":"chat.completion.chunk","created":1789000000,"model":"gpt-4o-mini","choices":[{"index":0,"delta":{"content":"It is currently "}}]} + + data: {"id":"chatcmpl-stream-3","object":"chat.completion.chunk","created":178900 + + """; + + private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper(); + + private static WireMockServer wireMockServer; + private static User user; + private static String bearerToken; + + private Host host; + private final ChatCompletionsResource resource = new ChatCompletionsResource(); + + @BeforeClass + public static void beforeClass() throws Exception { + IntegrationTestInitService.getInstance().init(); + IPUtils.disabledIpPrivateSubnet(true); + wireMockServer = AiTest.prepareWireMock(); + + // A bare UserDataGen user has no roles at all, so it is neither a backend nor a + // frontend user and FR-016 rejects it; it also cannot read the site these tests pass + // as an explicit override, which FR-019 checks. Two roles are needed, not one: the check + // in WebResource.checkRolePermissions is doesUserHaveRole(user, "DOTCMS_BACK_END_USER") + // by key and does not walk inheritance, so being an admin does not imply it. Admin is + // what grants read on the site. Role-specific behaviour is US3's tests, not these. + user = new UserDataGen() + .roles(APILocator.getRoleAPI().loadBackEndUserRole(), + APILocator.getRoleAPI().loadCMSAdminRole()) + .nextPersisted(); + final ApiToken apiToken = APILocator.getApiTokenAPI().persistApiToken( + user.getUserId(), + Date.from(Instant.now().plus(Duration.ofDays(1))), + APILocator.systemUser().getUserId(), + "127.0.0.1"); + bearerToken = "Bearer " + APILocator.getApiTokenAPI().getJWT(apiToken, user); + } + + @AfterClass + public static void afterClass() { + wireMockServer.stop(); + IPUtils.disabledIpPrivateSubnet(false); + } + + @Before + public void before() throws Exception { + host = new SiteDataGen().nextPersisted(); + AiTest.aiAppSecretsWithProviderConfig(host, AiTest.providerConfigJson(AiTest.PORT, CHAT_MODEL)); + wireMockServer.resetAll(); + } + + @After + public void after() throws Exception { + AiTest.removeAiAppSecrets(host); + } + + /** + * Given a provider streaming a plain content answer + * When the completion is requested with {@code stream: true} + * Then the body is a sequence of {@code data:} frames, every chunk carrying {@code id}, + * {@code object}, {@code created} and {@code model}, the last content chunk carrying a + * {@code finish_reason}, and the stream closing with the terminal marker + */ + @Test + public void test_completions_withStreamRequested_writesChunkFramesEndingWithDoneMarker() throws Exception { + stubProviderStream(PROVIDER_CONTENT_STREAM); + + final Response response = resource.completions( + mockRequest(), mockResponse(), host.getIdentifier(), streamingRequest(null)); + + assertNotNull(response); + assertEquals(200, response.getStatus()); + assertTrue(response.getEntity() instanceof StreamingOutput); + + final String body = drain(response); + assertTrue(body.startsWith(DATA_PREFIX)); + assertTrue(body.endsWith(SseSerializer.DONE_MARKER)); + + final ListDeliberately one stub, not a pair keyed on the outbound body: {@code stream_options} is + * an OpenAI-format option that four of dotCMS's seven providers do not understand, so + * forwarding it would make streamed usage work on some vendors and silently not on others. + * dotCMS instead reads the flag itself and emits the usage chunk from the token counts the + * unified provider abstraction already hands back on completion, which behaves identically + * whichever vendor served the request.
+ */ + private static void stubProviderStreamWithoutUsage() { + stubProviderStream(PROVIDER_CONTENT_STREAM); + } + + /** + * Stubs a provider that volunteers a usage chunk nobody asked for. + * + *Some providers do this. The chunk carries an empty {@code choices} array, which is + * exactly the shape that breaks stream readers assuming every chunk has a choice — so it must + * not reach a caller who did not ask for usage.
+ */ + private static void stubProviderStreamVolunteeringUsage() { + stubProviderStream(PROVIDER_CONTENT_STREAM_WITH_USAGE); + } + + /** + * @param streamOptions the streaming options to send, or null to send none + * @return a one-turn conversation asking for a streamed answer + */ + private static ChatCompletionRequestView streamingRequest(final StreamOptionsView streamOptions) { + return new ChatCompletionRequestView( + CHAT_MODEL, + List.of(userMessage("What is the weather in Bogota?")), + null, + null, + null, + Boolean.TRUE, + streamOptions, + null, null, null, null, null); + } + + /** + * Writes the streamed entity out in full, the way the container would. + * + * @param response the response the resource returned + * @return everything the caller would have read off the stream + */ + private static String drain(final Response response) throws Exception { + final ByteArrayOutputStream out = new ByteArrayOutputStream(); + ((StreamingOutput) response.getEntity()).write(out); + return out.toString(StandardCharsets.UTF_8); + } + + /** + * Whatever the caller ends up seeing, whether the failure was delivered as a stream or as a + * plain error body. Used only where the invariant under test holds for both. + * + * @param response the response the resource returned + * @return the response body, as text + */ + private static String clientVisibleBody(final Response response) throws Exception { + return response.getEntity() instanceof StreamingOutput + ? drain(response) + : OBJECT_MAPPER.writeValueAsString(response.getEntity()); + } + + /** + * @param body the streamed text + * @return every {@code data:} frame parsed as JSON, the terminal marker excluded + */ + private static ListThe round trip is the whole point of the tool contract and it cannot be verified one half at + * a time: a {@code tool_calls} answer is only useful if the id it carries is accepted back on the + * next turn, and an id that is accepted back is only safe if an uncorrelated one is refused. + * The four tests here cover exactly that arc.
+ * + *The provider is a WireMock server standing in for an OpenAI-compatible endpoint, wired in + * through the same dotAI app secrets the rest of the AI integration tests use, so the exchange + * travels the real client path rather than a stubbed one.
+ */ +public class ChatCompletionsTest { + + /** The chat model the site is configured with, and the only one these tests ask for. */ + private static final String CHAT_MODEL = "gpt-4o-mini"; + + /** Path an OpenAI-compatible provider serves completions on. */ + private static final String COMPLETIONS_PATH = "/chat/completions"; + + /** The tool the request declares and the stubbed provider asks to run. */ + private static final String TOOL_NAME = "get_weather"; + + /** The id the stubbed provider mints for its tool call, and that the follow-up replays. */ + private static final String TOOL_CALL_ID = "call_dot_weather_1"; + + /** An id no assistant turn in the conversation ever produced. */ + private static final String UNCORRELATED_TOOL_CALL_ID = "call_never_issued"; + + private static final String ERROR_TYPE_INVALID_REQUEST = "invalid_request_error"; + + /** What the provider answers when the conversation has not yet carried a tool result. */ + private static final String PROVIDER_TOOL_CALL_RESPONSE = """ + { + "id": "chatcmpl-tool-1", + "object": "chat.completion", + "created": 1789000000, + "model": "gpt-4o-mini", + "choices": [ + { + "index": 0, + "message": { + "role": "assistant", + "content": null, + "tool_calls": [ + { + "id": "call_dot_weather_1", + "type": "function", + "function": { + "name": "get_weather", + "arguments": "{\\"city\\":\\"Bogota\\"}" + } + } + ] + }, + "finish_reason": "tool_calls" + } + ], + "usage": {"prompt_tokens": 82, "completion_tokens": 17, "total_tokens": 99} + } + """; + + /** What the provider answers once the tool result is part of the conversation. */ + private static final String PROVIDER_FINAL_ANSWER_RESPONSE = """ + { + "id": "chatcmpl-tool-2", + "object": "chat.completion", + "created": 1789000001, + "model": "gpt-4o-mini", + "choices": [ + { + "index": 0, + "message": { + "role": "assistant", + "content": "It is currently 19 degrees Celsius in Bogota." + }, + "finish_reason": "stop" + } + ], + "usage": {"prompt_tokens": 120, "completion_tokens": 12, "total_tokens": 132} + } + """; + + private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper(); + + private static WireMockServer wireMockServer; + private static User user; + private static String bearerToken; + + private Host host; + private final ChatCompletionsResource resource = new ChatCompletionsResource(); + + @BeforeClass + public static void beforeClass() throws Exception { + IntegrationTestInitService.getInstance().init(); + IPUtils.disabledIpPrivateSubnet(true); + wireMockServer = AiTest.prepareWireMock(); + stubProvider(); + + // A bare UserDataGen user has no roles at all, so it is neither a backend nor a + // frontend user and FR-016 rejects it; it also cannot read the site these tests pass + // as an explicit override, which FR-019 checks. Two roles are needed, not one: the check + // in WebResource.checkRolePermissions is doesUserHaveRole(user, "DOTCMS_BACK_END_USER") + // by key and does not walk inheritance, so being an admin does not imply it. Admin is + // what grants read on the site. Role-specific behaviour is US3's tests, not these. + user = new UserDataGen() + .roles(APILocator.getRoleAPI().loadBackEndUserRole(), + APILocator.getRoleAPI().loadCMSAdminRole()) + .nextPersisted(); + final ApiToken apiToken = APILocator.getApiTokenAPI().persistApiToken( + user.getUserId(), + Date.from(Instant.now().plus(Duration.ofDays(1))), + APILocator.systemUser().getUserId(), + "127.0.0.1"); + bearerToken = "Bearer " + APILocator.getApiTokenAPI().getJWT(apiToken, user); + } + + @AfterClass + public static void afterClass() { + wireMockServer.stop(); + IPUtils.disabledIpPrivateSubnet(false); + } + + @Before + public void before() throws Exception { + host = new SiteDataGen().nextPersisted(); + AiTest.aiAppSecretsWithProviderConfig(host, AiTest.providerConfigJson(AiTest.PORT, CHAT_MODEL)); + wireMockServer.resetRequests(); + } + + @After + public void after() throws Exception { + AiTest.removeAiAppSecrets(host); + } + + /** + * Given a request declaring a {@code get_weather} tool, against a provider that answers with a + * tool call + * When the completion is requested + * Then the answer finishes for {@code tool_calls} and carries one tool call with an id, the + * declared function name and its arguments + */ + @Test + public void test_completions_withDeclaredTool_returnsToolCall() throws Exception { + final ChatCompletionRequestView requestView = new ChatCompletionRequestView( + CHAT_MODEL, + List.of( + systemMessage("You are a helpful assistant."), + userMessage("What is the weather in Bogota?")), + List.of(weatherTool()), + TextNode.valueOf("auto"), + null, null, null, null, null, null, null, null); + + final Response response = resource.completions( + mockRequest(), mockResponse(), host.getIdentifier(), requestView); + + assertNotNull(response); + assertEquals(200, response.getStatus()); + assertTrue(response.getEntity() instanceof ChatCompletionView); + + final ChatCompletionView view = (ChatCompletionView) response.getEntity(); + assertNotNull(view.choices()); + assertEquals(1, view.choices().size()); + + final ChatCompletionView.ChoiceView choice = view.choices().get(0); + assertEquals("tool_calls", choice.finishReason()); + assertNotNull(choice.message()); + assertNotNull(choice.message().toolCalls()); + assertEquals(1, choice.message().toolCalls().size()); + + final ChatCompletionView.ToolCallView toolCall = choice.message().toolCalls().get(0); + assertEquals(TOOL_CALL_ID, toolCall.id()); + assertNotNull(toolCall.function()); + assertEquals(TOOL_NAME, toolCall.function().name()); + assertNotNull(toolCall.function().arguments()); + assertFalse(toolCall.function().arguments().isBlank()); + assertTrue(toolCall.function().arguments().contains("Bogota")); + } + + /** + * Given a conversation replaying the assistant tool-call turn and a {@code role:"tool"} turn + * carrying that same {@code tool_call_id} + * When the completion is requested + * Then the turn is accepted and the provider's answer comes back as a finished completion + */ + @Test + public void test_completions_withCorrelatedToolResult_returnsAnswer() throws Exception { + final ChatCompletionRequestView requestView = new ChatCompletionRequestView( + CHAT_MODEL, + List.of( + userMessage("What is the weather in Bogota?"), + assistantToolCallMessage(TOOL_CALL_ID), + toolResultMessage(TOOL_CALL_ID, "{\"tempC\":19}")), + List.of(weatherTool()), + TextNode.valueOf("auto"), + null, null, null, null, null, null, null, null); + + final Response response = resource.completions( + mockRequest(), mockResponse(), host.getIdentifier(), requestView); + + assertNotNull(response); + assertEquals(200, response.getStatus()); + assertTrue(response.getEntity() instanceof ChatCompletionView); + + final ChatCompletionView view = (ChatCompletionView) response.getEntity(); + assertNotNull(view.choices()); + assertEquals(1, view.choices().size()); + + final ChatCompletionView.ChoiceView choice = view.choices().get(0); + assertEquals("stop", choice.finishReason()); + assertNotNull(choice.message()); + assertNotNull(choice.message().content()); + assertFalse(choice.message().content().isBlank()); + + wireMockServer.verify(1, postRequestedFor(urlPathEqualTo(COMPLETIONS_PATH)) + .withRequestBody(containing(TOOL_CALL_ID))); + } + + /** + * Given a {@code role:"tool"} turn whose {@code tool_call_id} matches no preceding assistant + * tool call + * When the completion is requested + * Then it is refused as a client error and the provider is never contacted + */ + @Test + public void test_completions_withUncorrelatedToolResult_isRejected() throws Exception { + final ChatCompletionRequestView requestView = new ChatCompletionRequestView( + CHAT_MODEL, + List.of( + userMessage("What is the weather in Bogota?"), + toolResultMessage(UNCORRELATED_TOOL_CALL_ID, "{\"tempC\":19}")), + List.of(weatherTool()), + TextNode.valueOf("auto"), + null, null, null, null, null, null, null, null); + + final Response response = resource.completions( + mockRequest(), mockResponse(), host.getIdentifier(), requestView); + + assertNotNull(response); + assertEquals(400, response.getStatus()); + assertTrue(response.getEntity() instanceof InferenceErrorView); + + final InferenceErrorView errorView = (InferenceErrorView) response.getEntity(); + assertNotNull(errorView.error()); + assertEquals(ERROR_TYPE_INVALID_REQUEST, errorView.error().type()); + assertNotNull(errorView.error().message()); + assertFalse(errorView.error().message().isBlank()); + + wireMockServer.verify(0, postRequestedFor(urlPathEqualTo(COMPLETIONS_PATH))); + } + + /** + * Given a request that omits {@code model} + * When the completion is requested + * Then it is refused as a client error naming {@code model}, with no implicit default applied + */ + @Test + public void test_completions_withoutModel_isRejected() { + final ChatCompletionRequestView requestView = new ChatCompletionRequestView( + null, + List.of(userMessage("What is the weather in Bogota?")), + null, null, null, null, null, null, null, null, null, null); + + final Response response = resource.completions( + mockRequest(), mockResponse(), host.getIdentifier(), requestView); + + assertNotNull(response); + assertEquals(400, response.getStatus()); + assertTrue(response.getEntity() instanceof InferenceErrorView); + + final InferenceErrorView errorView = (InferenceErrorView) response.getEntity(); + assertNotNull(errorView.error()); + assertEquals(ERROR_TYPE_INVALID_REQUEST, errorView.error().type()); + assertEquals("model", errorView.error().param()); + + wireMockServer.verify(0, postRequestedFor(urlPathEqualTo(COMPLETIONS_PATH))); + } + + /** + * Stubs the OpenAI-compatible provider. The tool-result stub takes precedence — a conversation + * carrying a tool result also carries the tool declaration, so ordering by priority is what + * separates the two turns. + */ + private static void stubProvider() { + wireMockServer.stubFor(post(urlPathEqualTo(COMPLETIONS_PATH)) + .atPriority(1) + .withRequestBody(containing(TOOL_CALL_ID)) + .willReturn(aResponse() + .withStatus(200) + .withHeader("Content-Type", "application/json") + .withBody(PROVIDER_FINAL_ANSWER_RESPONSE))); + + wireMockServer.stubFor(post(urlPathEqualTo(COMPLETIONS_PATH)) + .atPriority(2) + .withRequestBody(containing(TOOL_NAME)) + .willReturn(aResponse() + .withStatus(200) + .withHeader("Content-Type", "application/json") + .withBody(PROVIDER_TOOL_CALL_RESPONSE))); + } + + /** + * @return a request authenticated with the test user's bearer token, as the family requires + */ + private static HttpServletRequest mockRequest() { + final HttpServletRequest request = mock(HttpServletRequest.class); + when(request.getRequestURI()).thenReturn("/api/inference/v1/chat/completions"); + when(request.getRequestURL()) + .thenReturn(new StringBuffer("http://localhost/api/inference/v1/chat/completions")); + when(request.getMethod()).thenReturn("POST"); + when(request.getRemoteAddr()).thenReturn("127.0.0.1"); + when(request.getHeader("Authorization")).thenReturn(bearerToken); + return request; + } + + private static HttpServletResponse mockResponse() { + return mock(HttpServletResponse.class); + } + + private static MessageView systemMessage(final String content) { + return new MessageView("system", content, null, null, null); + } + + private static MessageView userMessage(final String content) { + return new MessageView("user", content, null, null, null); + } + + /** + * @param toolCallId the id the assistant turn asked the client to echo back + * @return the assistant turn replaying a tool call, exactly as a client would resend it + */ + private static MessageView assistantToolCallMessage(final String toolCallId) { + return new MessageView( + "assistant", + null, + List.of(new ToolCallView( + toolCallId, + "function", + new FunctionCallView(TOOL_NAME, "{\"city\":\"Bogota\"}"))), + null, + null); + } + + /** + * @param toolCallId the call this result answers + * @param content the tool's output, as JSON text + * @return the {@code role:"tool"} turn carrying the result + */ + private static MessageView toolResultMessage(final String toolCallId, final String content) { + return new MessageView("tool", content, null, toolCallId, TOOL_NAME); + } + + /** + * @return the {@code get_weather} tool declaration, with a JSON Schema for its arguments + */ + private static ToolView weatherTool() throws Exception { + final JsonNode parameters = OBJECT_MAPPER.readTree(""" + { + "type": "object", + "properties": {"city": {"type": "string"}}, + "required": ["city"] + } + """); + return new ToolView( + "function", + new FunctionDefView(TOOL_NAME, "Current weather for a city", parameters)); + } +} diff --git a/dotcms-integration/src/test/java/com/dotcms/inference/rest/InferenceAuthorizationTest.java b/dotcms-integration/src/test/java/com/dotcms/inference/rest/InferenceAuthorizationTest.java new file mode 100644 index 000000000000..ddf12b6b08fd --- /dev/null +++ b/dotcms-integration/src/test/java/com/dotcms/inference/rest/InferenceAuthorizationTest.java @@ -0,0 +1,383 @@ +package com.dotcms.inference.rest; + +import com.dotcms.ai.AiTest; +import com.dotcms.auth.providers.jwt.beans.ApiToken; +import com.dotcms.datagen.SiteDataGen; +import com.dotcms.datagen.UserDataGen; +import com.dotcms.inference.rest.view.ChatCompletionRequestView; +import com.dotcms.inference.rest.view.ChatCompletionRequestView.MessageView; +import com.dotcms.inference.rest.view.ChatCompletionView; +import com.dotcms.inference.rest.view.InferenceErrorView; +import com.dotcms.util.IntegrationTestInitService; +import com.dotcms.util.network.IPUtils; +import com.dotmarketing.beans.Host; +import com.dotmarketing.business.APILocator; +import com.github.tomakehurst.wiremock.WireMockServer; +import com.liferay.portal.model.User; +import org.junit.After; +import org.junit.AfterClass; +import org.junit.Before; +import org.junit.BeforeClass; +import org.junit.Test; + +import javax.servlet.http.HttpServletRequest; +import javax.servlet.http.HttpServletResponse; +import javax.servlet.http.HttpSession; +import javax.ws.rs.WebApplicationException; +import javax.ws.rs.core.Response; +import java.time.Duration; +import java.time.Instant; +import java.util.Date; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.UUID; + +import static com.github.tomakehurst.wiremock.client.WireMock.aResponse; +import static com.github.tomakehurst.wiremock.client.WireMock.post; +import static com.github.tomakehurst.wiremock.client.WireMock.urlPathEqualTo; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.anyString; +import static org.mockito.Mockito.doAnswer; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; + +/** + * Specifies who may call {@code /api/inference/v1} and on whose behalf — the authentication and + * site-authorization half of User Story 3. + * + *The three requirements pull in different directions on purpose, and each is only meaningful + * next to the others:
+ * + *Refusals arrive in two different shapes, which is a property of the contract rather than an + * inconsistency. An authentication failure is raised by {@code WebResource} before the resource + * body runs, so it surfaces as a {@link WebApplicationException} carrying the status; a site + * authorization failure is caught by the resource and rendered as an {@link InferenceErrorView}, + * so that a standard client can deserialize it with no adapter.
+ */ +public class InferenceAuthorizationTest { + + /** Path an OpenAI-compatible provider serves completions on. */ + private static final String COMPLETIONS_PATH = "/chat/completions"; + + private static final String REQUEST_URI = "/api/inference/v1/chat/completions"; + + /** The chat model the site under test is configured with. */ + private static final String CHAT_MODEL = "gpt-4o-mini"; + + private static final String ERROR_TYPE_INVALID_REQUEST = "invalid_request_error"; + + /** What the stubbed provider answers whenever a call gets that far. */ + private static final String PROVIDER_RESPONSE = """ + { + "id": "chatcmpl-auth-1", + "object": "chat.completion", + "created": 1789000000, + "model": "gpt-4o-mini", + "choices": [ + { + "index": 0, + "message": {"role": "assistant", "content": "Answered."}, + "finish_reason": "stop" + } + ], + "usage": {"prompt_tokens": 10, "completion_tokens": 2, "total_tokens": 12} + } + """; + + private static WireMockServer wireMockServer; + + /** A backend user who is also an administrator, so site READ is never the thing under test. */ + private static User backendUser; + private static String backendBearerToken; + + /** A frontend-only user: authenticated, unprivileged, and admitted by FR-016 all the same. */ + private static User frontendUser; + private static String frontendBearerToken; + + /** A backend user who is not an administrator, and so cannot read an arbitrary site. */ + private static User unprivilegedUser; + private static String unprivilegedBearerToken; + + private Host host; + + private final ChatCompletionsResource resource = new ChatCompletionsResource(); + + @BeforeClass + public static void beforeClass() throws Exception { + IntegrationTestInitService.getInstance().init(); + IPUtils.disabledIpPrivateSubnet(true); + wireMockServer = AiTest.prepareWireMock(); + stubProvider(); + + // Two roles, not one. WebResource.checkRolePermissions matches DOTCMS_BACK_END_USER by + // key and does not walk inheritance, so being an administrator does not imply it; the + // administrator role is separately what grants READ on a site. + backendUser = new UserDataGen() + .roles(APILocator.getRoleAPI().loadBackEndUserRole(), + APILocator.getRoleAPI().loadCMSAdminRole()) + .nextPersisted(); + backendBearerToken = bearerTokenFor(backendUser); + + frontendUser = new UserDataGen() + .roles(APILocator.getRoleAPI().loadFrontEndUserRole()) + .nextPersisted(); + frontendBearerToken = bearerTokenFor(frontendUser); + + // Deliberately no administrator role: this user passes authentication and the role gate, + // and then fails on site READ, which is exactly the boundary FR-019 draws. + unprivilegedUser = new UserDataGen() + .roles(APILocator.getRoleAPI().loadBackEndUserRole()) + .nextPersisted(); + unprivilegedBearerToken = bearerTokenFor(unprivilegedUser); + } + + @AfterClass + public static void afterClass() { + wireMockServer.stop(); + IPUtils.disabledIpPrivateSubnet(false); + } + + @Before + public void before() throws Exception { + host = new SiteDataGen() + .name("inference-auth-" + UUID.randomUUID() + ".dotcms.com") + .nextPersisted(); + AiTest.aiAppSecretsWithProviderConfig( + host, AiTest.providerConfigJson(AiTest.PORT, CHAT_MODEL)); + wireMockServer.resetRequests(); + } + + @After + public void after() throws Exception { + AiTest.removeAiAppSecrets(host); + } + + /** + * Given a request carrying no credential at all + * When a completion is requested + * Then it is refused as unauthorized + * + *FR-015. The refusal carries the standard error shape, as the contract's status table + * requires of a 401 — a caller's client library should be able to deserialize a refusal into + * its own error type exactly as it would a success. Accepts a thrown + * {@link WebApplicationException} too, since the surrounding authentication handshake may + * refuse before the resource body runs and either is a valid refusal.
+ */ + @Test + public void test_completions_withNoCredential_isUnauthorized() { + final HttpServletRequest request = mockRequest(host.getHostname(), null); + + try { + final Response response = + resource.completions(request, mockResponse(), null, completionFor(CHAT_MODEL)); + assertEquals("An anonymous request must be refused as unauthorized", + Response.Status.UNAUTHORIZED.getStatusCode(), response.getStatus()); + assertTrue("a refusal carries the standard error shape", + response.getEntity() instanceof InferenceErrorView); + } catch (final WebApplicationException e) { + assertEquals(Response.Status.UNAUTHORIZED.getStatusCode(), + e.getResponse().getStatus()); + } + } + + /** + * Given a request authenticated by a session rather than by a bearer token, and otherwise + * entirely valid + * When a completion is requested + * Then it is refused as unauthorized + * + *FR-015 accepts a dotCMS API token presented as a bearer credential and nothing else. The + * session cookies the surrounding dotCMS filters generally honour must not be honoured here: + * this family is server-side only by design, and a browser that can be made to carry an + * ambient session would otherwise be able to spend a site's AI credentials.
+ */ + @Test + public void test_completions_withSessionCredentialOnly_isUnauthorized() { + final HttpServletRequest request = mockRequest(host.getHostname(), null); + givenSessionAuthenticatedAs(request, backendUser); + + try { + final Response response = resource.completions( + request, mockResponse(), null, completionFor(CHAT_MODEL)); + assertEquals("A session credential must not authenticate this family", + Response.Status.UNAUTHORIZED.getStatusCode(), response.getStatus()); + assertTrue("a refusal carries the standard error shape", + response.getEntity() instanceof InferenceErrorView); + } catch (final WebApplicationException e) { + assertEquals(Response.Status.UNAUTHORIZED.getStatusCode(), + e.getResponse().getStatus()); + } + } + + /** + * Given an authenticated caller who cannot read the site they name explicitly + * When a completion is requested with that site as the override + * Then it is refused as forbidden, in the standard error shape, naming the offending field + * + *FR-019. The caller authenticates and clears the role gate; what stops them is READ on the + * site whose credentials they asked to spend.
+ */ + @Test + public void test_completions_withOverrideForUnreadableSite_isForbidden() { + final HttpServletRequest request = + mockRequest(host.getHostname(), unprivilegedBearerToken); + + final Response response = resource.completions( + request, mockResponse(), host.getIdentifier(), completionFor(CHAT_MODEL)); + + assertNotNull(response); + assertEquals(Response.Status.FORBIDDEN.getStatusCode(), response.getStatus()); + assertTrue(response.getEntity() instanceof InferenceErrorView); + + final InferenceErrorView errorView = (InferenceErrorView) response.getEntity(); + assertNotNull(errorView.error()); + assertEquals(ERROR_TYPE_INVALID_REQUEST, errorView.error().type()); + assertEquals("siteId", errorView.error().param()); + } + + /** + * Given a backend user authenticated by bearer token + * When a completion is requested + * Then it is served + * + *FR-016, one half. Stated alongside the frontend case because the requirement is that + * both are admitted; either one alone would be satisfied by a rule that excluded the + * other.
+ */ + @Test + public void test_completions_withBackendUser_isAccepted() { + final HttpServletRequest request = mockRequest(host.getHostname(), backendBearerToken); + + final Response response = resource.completions( + request, mockResponse(), null, completionFor(CHAT_MODEL)); + + assertNotNull(response); + assertEquals(200, response.getStatus()); + assertTrue(response.getEntity() instanceof ChatCompletionView); + } + + /** + * Given a frontend user authenticated by bearer token + * When a completion is requested + * Then it is served + * + *FR-016, the other half. A site calling AI on behalf of a visitor is a supported use case, + * so the bar on this family is authentication, not privilege.
+ */ + @Test + public void test_completions_withFrontendUser_isAccepted() { + final HttpServletRequest request = mockRequest(host.getHostname(), frontendBearerToken); + + final Response response = resource.completions( + request, mockResponse(), null, completionFor(CHAT_MODEL)); + + assertNotNull(response); + assertEquals(200, response.getStatus()); + assertTrue(response.getEntity() instanceof ChatCompletionView); + } + + /** Answers every completion that gets as far as the provider with the same finished response. */ + private static void stubProvider() { + wireMockServer.stubFor(post(urlPathEqualTo(COMPLETIONS_PATH)) + .willReturn(aResponse() + .withStatus(200) + .withHeader("Content-Type", "application/json") + .withBody(PROVIDER_RESPONSE))); + } + + /** + * @param user the caller to mint a credential for + * @return the {@code Authorization} header value carrying that caller's API token + */ + private static String bearerTokenFor(final User user) throws Exception { + final ApiToken apiToken = APILocator.getApiTokenAPI().persistApiToken( + user.getUserId(), + Date.from(Instant.now().plus(Duration.ofDays(1))), + APILocator.systemUser().getUserId(), + "127.0.0.1"); + return "Bearer " + APILocator.getApiTokenAPI().getJWT(apiToken, user); + } + + /** + * @param model the model to ask for + * @return the smallest well-formed completion request + */ + private static ChatCompletionRequestView completionFor(final String model) { + return new ChatCompletionRequestView( + model, + List.of(new MessageView("user", "May I call you?", null, null, null)), + null, null, null, null, null, null, null, null, null, null); + } + + /** + * Gives the request a logged-in session, the way a browser carrying a dotCMS session cookie + * would present itself to the servlet layer. + * + * @param request the request to attach the session to + * @param user the user the session is logged in as + */ + private static void givenSessionAuthenticatedAs(final HttpServletRequest request, + final User user) { + final HttpSession session = mock(HttpSession.class); + when(session.getAttribute(com.liferay.portal.util.WebKeys.USER)).thenReturn(user); + when(session.getAttribute(com.liferay.portal.util.WebKeys.USER_ID)) + .thenReturn(user.getUserId()); + when(request.getSession(false)).thenReturn(session); + when(request.getSession()).thenReturn(session); + } + + /** + * Builds a request arriving at a given host name, with or without a bearer credential. + * + *Request attributes are backed by a real map rather than left as mock no-ops, because the + * authentication handshake publishes the resolved caller as an attribute and a mock that + * forgot it would not behave like a servlet container.
+ * + * @param serverName the host name the request arrives on + * @param bearerToken the {@code Authorization} header value, or null for no credential + * @return the mocked request + */ + private static HttpServletRequest mockRequest(final String serverName, + final String bearerToken) { + final HttpServletRequest request = mock(HttpServletRequest.class); + final MapFR-011 is not "expose an embeddings endpoint" — the vector round trip is the easy half. What + * the requirement actually adds is that the model gate of FR-023 has to look at a + * different section of the site's configuration than the chat endpoint does. A + * site configures its chat models and its embeddings model separately, so an implementation that + * reuses the chat section here would accept a chat model for an embeddings call and refuse the + * embeddings model the site actually configured, and it would do so while returning a perfectly + * well-shaped 200 for the wrong request. That is why the central test below asserts both halves of + * the same site in one go: the embeddings model succeeds and the chat model is refused. + * Either assertion alone is satisfied by the bug.
+ * + *The second thing FR-011 adds is that {@code input} is not a string. It is either a + * string or an array of strings, because batching is how content is ordinarily embedded — + * anyone indexing a site sends an array — and a scalar-only endpoint would fail that common case + * while still looking correct against a one-off example. The array form is what makes the response + * a list of more than one entry, and therefore what makes {@code index} load-bearing: it is the + * only thing that lets a caller correlate a vector back to the text it sent. The tests below + * assert that index explicitly, looking each entry up by it rather than by its position in + * {@code data}, and they pin every refusal the array form introduces — nothing to embed, and + * elements that are not text.
+ * + *FR-011 now spells out what "an array of strings" costs an implementation, and each clause is + * a test below. Every element has to be a string, so the all-numeric array is + * joined by a mixed one: an implementation that inspects only the first element passes the + * all-numeric case and still accepts {@code ["some text", 5]}, which is the shape that actually + * arrives when a caller's collection was built from mismatched sources. An absent, null or + * empty input is refused naming the field rather than answered with an empty list, and + * "null" has two wire forms — the JSON null and the field left out — which are separate code paths + * and so are asserted separately.
+ * + *FR-011 also settles two things this file used to leave open. A blank or + * whitespace-only string is refused on the same grounds as an absent, null or empty one, + * whether it arrives alone or as one element of an array: there is no meaningful embedding of + * nothing, and providers differ in whether they error on it or hand back a zero vector — which is + * exactly the inconsistency this family exists to hide, so it cannot be left to whichever provider + * a site configured. Both positions are asserted, because they are different code paths: the scalar + * blank is caught by the same guard that catches the empty string, while a blank buried at + * position two of an array is only caught by an implementation that inspects every element.
+ * + *And when one element of an array is at fault, {@code param} stays {@code input} — it is the + * field the caller sent — but the message has to identify which element. + * A caller who batched five hundred strings and is told only that "input is invalid" has been + * handed the search, not the answer. The tests below therefore assert that the offending position + * appears in the message, rather than settling for a message that is merely non-blank; the + * offending element is deliberately placed at an index that appears nowhere else in the payload, + * so a message that happens to contain a digit cannot pass for one that locates the fault.
+ * + *The provider is a WireMock server standing in for an OpenAI-compatible endpoint, wired in + * through the same dotAI app secrets the rest of the AI integration tests use, so an accepted call + * travels the real client path rather than a stubbed one.
+ */ +public class InferenceEmbeddingsTest { + + /** + * The model the site's {@code embeddings} section is configured with. Deliberately read from + * {@link AiTest} rather than restated: the separation this file is about is a property of that + * shared configuration, and a local copy would keep passing if the two drifted apart. + */ + private static final String EMBEDDINGS_MODEL = AiTest.EMBEDDINGS_MODEL; + + /** + * The model the same site's {@code chat} section is configured with — configured, valid, and + * nevertheless not an embeddings model. + */ + private static final String CHAT_MODEL = "gpt-4o-mini"; + + /** A model name no site in these tests configures for any section at all. */ + private static final String UNCONFIGURED_MODEL = "some-other-vendors-model"; + + /** Path an OpenAI-compatible provider serves embeddings on. */ + private static final String EMBEDDINGS_PATH = "/embeddings"; + + private static final String REQUEST_URI = "/api/inference/v1/embeddings"; + + private static final String ERROR_TYPE_INVALID_REQUEST = "invalid_request_error"; + + /** The text these tests embed. */ + private static final String INPUT_TEXT = "The quick brown fox"; + + /** A string with nothing in it at all — the emptiest thing a caller can ask to embed. */ + private static final String EMPTY_INPUT = ""; + + /** + * A string with nothing in it but whitespace. FR-011 refuses it on the same grounds as the + * empty string, and it is asserted separately because it is a separate guard: an + * {@code isEmpty()} check accepts it and passes three spaces to a provider that will either + * error or bill for a zero vector. + */ + private static final String WHITESPACE_INPUT = " "; + + /** + * The value that stands in for "not a string" inside a mixed array. Two digits, neither of + * which is the index the element sits at, so that a message quoting the offending value cannot + * be mistaken for one that locates it. + */ + private static final int NON_STRING_ELEMENT = 42; + + /** + * The batch these tests embed when they exercise the array form. Three entries rather than two, + * so that "one entry per input" cannot be confused with "a pair", and the last one is the + * marker the batch stub matches on — see {@link #stubProvider()}. + */ + private static final String[] BATCH_INPUT = { + INPUT_TEXT, + "jumps over the lazy dog", + "and the dog barks back" + }; + + /** + * The last element of {@link #BATCH_INPUT}, which appears in the provider request only when a + * batch was sent. Distinct enough that no scalar request can contain it by accident. + */ + private static final String BATCH_MARKER = BATCH_INPUT[BATCH_INPUT.length - 1]; + + /** Builds the array form of {@code input}; no configuration of its own is needed. */ + private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper(); + + /** + * What the stubbed provider answers. Modelled on what an OpenAI-compatible provider really + * returns — the shape of + * {@code dotcms-integration/src/test/resources/mappings/langchain4j-embeddings-stub.json}, + * trimmed to a vector short enough to read. + */ + private static final String PROVIDER_RESPONSE = """ + { + "object": "list", + "data": [ + { + "object": "embedding", + "index": 0, + "embedding": [0.0023, -0.0091, 0.0117, -0.0042, 0.0008] + } + ], + "model": "text-embedding-ada-002", + "usage": {"prompt_tokens": 5, "total_tokens": 5} + } + """; + + /** + * What the stubbed provider answers a batch of {@link #BATCH_INPUT} with: one entry per input, + * each with its own vector, and a token count covering all three rather than one of them. + * + *The vectors differ from one another on purpose. Three copies of the same numbers would + * still satisfy "three entries with three indexes" while the implementation handed every + * caller the first vector.
+ */ + private static final String PROVIDER_BATCH_RESPONSE = """ + { + "object": "list", + "data": [ + { + "object": "embedding", + "index": 0, + "embedding": [0.0023, -0.0091, 0.0117, -0.0042, 0.0008] + }, + { + "object": "embedding", + "index": 1, + "embedding": [0.0512, 0.0034, -0.0077, 0.0101, -0.0013] + }, + { + "object": "embedding", + "index": 2, + "embedding": [-0.0308, 0.0146, 0.0052, -0.0090, 0.0027] + } + ], + "model": "text-embedding-ada-002", + "usage": {"prompt_tokens": 17, "total_tokens": 17} + } + """; + + private static WireMockServer wireMockServer; + + /** A backend user who is also an administrator, so site READ is never the thing under test. */ + private static User user; + private static String bearerToken; + + private Host host; + private final EmbeddingsResource resource = new EmbeddingsResource(); + + @BeforeClass + public static void beforeClass() throws Exception { + IntegrationTestInitService.getInstance().init(); + IPUtils.disabledIpPrivateSubnet(true); + wireMockServer = AiTest.prepareWireMock(); + stubProvider(); + + // A bare UserDataGen user has no roles at all, so it is neither a backend nor a frontend + // user and FR-016 rejects it; it also cannot read the site these tests pass as an explicit + // override, which FR-019 checks. Two roles are needed, not one: the check in + // WebResource.checkRolePermissions is doesUserHaveRole(user, "DOTCMS_BACK_END_USER") by key + // and does not walk inheritance, so being an admin does not imply it. Admin is what grants + // read on the site. Role-specific behaviour is US3's tests, not these. + user = new UserDataGen() + .roles(APILocator.getRoleAPI().loadBackEndUserRole(), + APILocator.getRoleAPI().loadCMSAdminRole()) + .nextPersisted(); + final ApiToken apiToken = APILocator.getApiTokenAPI().persistApiToken( + user.getUserId(), + Date.from(Instant.now().plus(Duration.ofDays(1))), + APILocator.systemUser().getUserId(), + "127.0.0.1"); + bearerToken = "Bearer " + APILocator.getApiTokenAPI().getJWT(apiToken, user); + } + + @AfterClass + public static void afterClass() { + wireMockServer.stop(); + IPUtils.disabledIpPrivateSubnet(false); + } + + @Before + public void before() throws Exception { + host = new SiteDataGen() + .name("inference-embeddings-" + UUID.randomUUID() + ".dotcms.com") + .nextPersisted(); + // Configures chat, embeddings and image as three separate sections, each with its own + // model. The chat model given here is the one this file expects the embeddings gate to + // refuse. + AiTest.aiAppSecretsWithProviderConfig( + host, AiTest.providerConfigJson(AiTest.PORT, CHAT_MODEL)); + wireMockServer.resetRequests(); + } + + @After + public void after() throws Exception { + AiTest.removeAiAppSecrets(host); + } + + /** + * Given a site configured with an embeddings model, and a request carrying input text + * When the embedding is requested + * Then a vector comes back in the standard embeddings shape — a {@code list} of one + * {@code embedding} at index 0 — alongside the serving model and the token counts + */ + @Test + public void test_embeddings_withInputText_returnsVectorInStandardShape() { + final Response response = resource.embeddings( + mockRequest(host.getHostname(), bearerToken), mockResponse(), + host.getIdentifier(), embeddingFor(EMBEDDINGS_MODEL, INPUT_TEXT)); + + assertNotNull(response); + assertEquals(200, response.getStatus()); + assertTrue(response.getEntity() instanceof EmbeddingListView); + + final EmbeddingListView view = (EmbeddingListView) response.getEntity(); + assertEquals("list", view.object()); + assertEquals(EMBEDDINGS_MODEL, view.model()); + + assertNotNull(view.data()); + assertEquals(1, view.data().size()); + + final EmbeddingListView.EmbeddingView embedding = view.data().get(0); + assertEquals("embedding", embedding.object()); + assertEquals(0, embedding.index()); + assertNotNull(embedding.embedding()); + assertFalse("An embedding with no numbers in it is not an embedding", + embedding.embedding().isEmpty()); + + assertNotNull(view.usage()); + assertNotNull(view.usage().promptTokens()); + assertTrue(view.usage().promptTokens() > 0); + assertNotNull(view.usage().totalTokens()); + assertTrue(view.usage().totalTokens() > 0); + + wireMockServer.verify(1, postRequestedFor(urlPathEqualTo(EMBEDDINGS_PATH))); + } + + /** + * Given a request whose {@code input} is an array of three strings + * When the embeddings are requested + * Then three entries come back, one per input, each carrying the index of the input it + * corresponds to and a vector of its own + * + *FR-011's array form. The assertions look each entry up by its index rather than + * reading positions out of {@code data}, because the index is the whole mechanism by which a + * caller correlates a vector back to the text it sent; a test that trusted array position + * would pass against an implementation that stamped indexes on positionally and never + * populated them meaningfully at all.
+ */ + @Test + public void test_embeddings_withArrayInput_returnsOneEntryPerInputInOrder() { + final Response response = resource.embeddings( + mockRequest(host.getHostname(), bearerToken), mockResponse(), + host.getIdentifier(), embeddingForAll(EMBEDDINGS_MODEL, BATCH_INPUT)); + + assertNotNull(response); + assertEquals(200, response.getStatus()); + assertTrue(response.getEntity() instanceof EmbeddingListView); + + final EmbeddingListView view = (EmbeddingListView) response.getEntity(); + assertEquals("list", view.object()); + assertEquals(EMBEDDINGS_MODEL, view.model()); + + assertNotNull(view.data()); + assertEquals("An array of three inputs is three embeddings, not one", + BATCH_INPUT.length, view.data().size()); + + for (int index = 0; index < BATCH_INPUT.length; index++) { + final EmbeddingListView.EmbeddingView embedding = embeddingAtIndex(view, index); + assertNotNull("No entry carries index " + index + ", so the caller cannot tell which " + + "input it embedded", embedding); + assertEquals("embedding", embedding.object()); + assertNotNull(embedding.embedding()); + assertFalse("An embedding with no numbers in it is not an embedding", + embedding.embedding().isEmpty()); + } + + assertEquals("Every entry must carry a distinct vector; identical vectors would mean the " + + "same input was embedded three times", + BATCH_INPUT.length, + view.data().stream().map(EmbeddingListView.EmbeddingView::embedding) + .distinct().count()); + + // One batch is one provider round trip, not one per input. + wireMockServer.verify(1, postRequestedFor(urlPathEqualTo(EMBEDDINGS_PATH))); + } + + /** + * Given a request whose {@code input} is a single string rather than an array + * When the embedding is requested + * Then exactly one entry comes back, at index 0 + * + *The scalar form is the half of FR-011 that already worked, and widening {@code input} to + * accept an array is exactly the kind of change that quietly breaks it — by normalising the + * scalar into a one-element array and then losing it, or by rejecting anything that is not an + * array. Both halves of "a string or an array of strings" have to hold at once, so the scalar + * form gets its own guard rather than living only inside the standard-shape test above.
+ */ + @Test + public void test_embeddings_withScalarInput_returnsSingleEntryAtIndexZero() { + final Response response = resource.embeddings( + mockRequest(host.getHostname(), bearerToken), mockResponse(), + host.getIdentifier(), embeddingFor(EMBEDDINGS_MODEL, INPUT_TEXT)); + + assertNotNull(response); + assertEquals("A single string is still valid input, not a malformed array", + 200, response.getStatus()); + assertTrue(response.getEntity() instanceof EmbeddingListView); + + final EmbeddingListView view = (EmbeddingListView) response.getEntity(); + assertNotNull(view.data()); + assertEquals("One input is one embedding", 1, view.data().size()); + + final EmbeddingListView.EmbeddingView embedding = view.data().get(0); + assertEquals("embedding", embedding.object()); + assertEquals("The only entry of a scalar request sits at index 0", 0, embedding.index()); + assertNotNull(embedding.embedding()); + assertFalse(embedding.embedding().isEmpty()); + + wireMockServer.verify(1, postRequestedFor(urlPathEqualTo(EMBEDDINGS_PATH))); + } + + /** + * Given a request whose {@code input} is an empty array + * When the embeddings are requested + * Then it is refused as a client error naming {@code input}, and the provider is never + * contacted + * + *Embedding nothing is a caller mistake — a batch built from a query that matched no + * content, most likely. Answering it with an empty {@code data} list and a 200 would look like + * success, and the caller would index nothing and never find out why. FR-011 now says this in + * as many words — an absent, null or empty input is refused naming the field — where before it + * had to be inferred from the response being "one entry per input"; the test is unchanged, + * which is the point of recording it.
+ */ + @Test + public void test_embeddings_withEmptyArrayInput_isRejectedNamingTheField() { + final Response response = resource.embeddings( + mockRequest(host.getHostname(), bearerToken), mockResponse(), + host.getIdentifier(), embeddingForAll(EMBEDDINGS_MODEL)); + + assertNotNull(response); + assertEquals(400, response.getStatus()); + assertTrue("An empty batch must be refused, not answered with an empty list", + response.getEntity() instanceof InferenceErrorView); + + final InferenceErrorView.Body error = ((InferenceErrorView) response.getEntity()).error(); + assertNotNull(error); + assertEquals(ERROR_TYPE_INVALID_REQUEST, error.type()); + assertEquals("input", error.param()); + assertNotNull(error.message()); + assertFalse(error.message().isBlank()); + + wireMockServer.verify(0, postRequestedFor(urlPathEqualTo(EMBEDDINGS_PATH))); + } + + /** + * Given a request whose {@code input} array holds numbers rather than strings + * When the embeddings are requested + * Then it is refused as a client error naming {@code input}, and the provider is never + * contacted + * + *The format this family follows also accepts arrays of token ids, and dotCMS does not. + * That difference has to surface here, as a validation error pointing at {@code input}, rather + * than several layers down as whatever the provider client makes of a list of integers — a + * caller who sent pre-tokenized input deserves to be told that, not a deserialization + * stacktrace.
+ */ + @Test + public void test_embeddings_withNonStringArrayElements_isRejectedNamingTheField() { + final ArrayNode tokenIds = OBJECT_MAPPER.createArrayNode(); + tokenIds.add(1); + tokenIds.add(2); + tokenIds.add(3); + + assertInputRefusedNamingTheField(tokenIds); + } + + /** + * Given a request whose {@code input} array holds three strings followed by a number + * When the embeddings are requested + * Then it is refused as a client error naming {@code input}, and the message identifies the + * element at fault, and the provider is never contacted + * + *FR-011 requires every element of the array to be a string, and this is + * the test that makes that word load-bearing. The all-numeric array above is refused by an + * implementation that looks only at {@code input.get(0)}; a mixed array is not, and a mixed + * array is the one a caller actually sends — a list assembled from two sources where one + * yielded identifiers instead of text. The refusal has to happen here, naming the field, for + * the same reason as the all-numeric case: several layers down it becomes whatever the + * provider client makes of a heterogeneous list. The number sits last rather than second so + * that the strings ahead of it are not decoration: an implementation that checks the first two + * elements and stops passes a two-element mixed array and fails this one.
+ * + *Naming the field is no longer enough on its own. FR-011 requires the message to identify + * which element is at fault, because {@code param} can only ever say {@code input} and + * a caller who sent a batch is otherwise left to find the bad one themselves. The assertion + * looks for the offending position in the message; the position is one no other number in the + * payload shares, so a message that merely quotes the value it choked on does not pass for one + * that locates it.
+ * + *The provider check is not a formality either. An implementation that "cleans" the array + * by keeping the strings it recognises would send a perfectly valid batch upstream and answer + * 200 — silently embedding less than the caller asked for and shifting every index they meant + * to correlate on.
+ */ + @Test + public void test_embeddings_withMixedArrayElements_isRejectedIdentifyingTheElement() { + final ArrayNode mixed = OBJECT_MAPPER.createArrayNode(); + for (final String text : BATCH_INPUT) { + mixed.add(text); + } + mixed.add(NON_STRING_ELEMENT); + + assertInputRefusedIdentifyingTheElement(mixed, BATCH_INPUT.length); + } + + /** + * Given a request whose {@code input} is a blank string — first empty, then whitespace only + * When the embeddings are requested + * Then both are refused as a client error naming {@code input}, and the provider is never + * contacted + * + *FR-011 refuses a blank or whitespace-only string on the same grounds as an absent, null or + * empty input: there is nothing there to embed. Leaving it to the provider is the failure this + * family exists to prevent — some error, some return a zero vector, and a caller indexing a + * site would silently store a meaningless vector against a document and retrieve it forever + * after. The two forms are asserted together because they are one requirement and two guards: + * {@code isEmpty()} catches the first and waves the second through.
+ */ + @Test + public void test_embeddings_withBlankStringInput_isRejectedNamingTheField() { + assertInputRefusedNamingTheField(TextNode.valueOf(EMPTY_INPUT)); + assertInputRefusedNamingTheField(TextNode.valueOf(WHITESPACE_INPUT)); + } + + /** + * Given a request whose {@code input} array holds two strings and then a whitespace-only one + * When the embeddings are requested + * Then it is refused as a client error naming {@code input}, and the message identifies the + * element at fault, and the provider is never contacted + * + *The array position of the blank-string rule, and the harder half of it. A blank arriving + * alone is the caller's whole request and hard to miss; a blank buried in a batch is what + * actually happens — a column that was empty for one row of five hundred — and an + * implementation that validates the array as a whole rather than element by element sends it + * upstream without noticing. Refusing the batch rather than skipping the blank is the point: + * dropping it would shift the index of every entry after it, and FR-011 makes those indexes the + * caller's only means of correlating vectors back to what they sent.
+ * + *As with the mixed array, the message must say which element — and the blank sits at a + * position that appears nowhere else in the payload, so nothing but locating it will do.
+ */ + @Test + public void test_embeddings_withBlankArrayElement_isRejectedIdentifyingTheElement() { + final ArrayNode withBlank = OBJECT_MAPPER.createArrayNode(); + withBlank.add(BATCH_INPUT[0]); + withBlank.add(BATCH_INPUT[1]); + withBlank.add(WHITESPACE_INPUT); + + assertInputRefusedIdentifyingTheElement(withBlank, 2); + } + + /** + * Given a request whose {@code input} is null — first as a JSON null, then as a field left out + * of the payload entirely + * When the embeddings are requested + * Then both are refused as a client error naming {@code input}, and the provider is never + * contacted + * + *FR-011 lists "absent" and "null" alongside "empty", and the two are asserted together + * because they are not the same code path. A JSON {@code "input": null} deserializes to a + * {@link NullNode} — an object, present, and non-null as far as a {@code != null} guard is + * concerned — while an omitted field leaves the component null outright. An implementation + * guarding on one reaches the provider with the other, where a null input is either an empty + * batch answered as success or a stacktrace, neither of which tells the caller they sent + * nothing.
+ */ + @Test + public void test_embeddings_withNullInput_isRejectedNamingTheField() { + assertInputRefusedNamingTheField(NullNode.getInstance()); + assertInputRefusedNamingTheField(null); + } + + /** + * Given the same site embedding one short string and then a batch of three + * When the reported usage of each is compared + * Then the batch reports more prompt tokens than the single string did + * + *Usage is what the caller is billed on, so it has to describe the whole batch rather than + * whichever element happened to be counted. The assertion is relative — the batch is larger + * than the single string — rather than a fixed number: the count is the provider's to report, + * and pinning it would make this test a statement about the stub instead of about the + * endpoint.
+ */ + @Test + public void test_embeddings_withArrayInput_reportsUsageForWholeBatch() { + final Response single = resource.embeddings( + mockRequest(host.getHostname(), bearerToken), mockResponse(), + host.getIdentifier(), embeddingFor(EMBEDDINGS_MODEL, INPUT_TEXT)); + + assertNotNull(single); + assertEquals(200, single.getStatus()); + assertTrue(single.getEntity() instanceof EmbeddingListView); + + final EmbeddingListView singleView = (EmbeddingListView) single.getEntity(); + assertNotNull(singleView.usage()); + assertNotNull(singleView.usage().promptTokens()); + + final Response batch = resource.embeddings( + mockRequest(host.getHostname(), bearerToken), mockResponse(), + host.getIdentifier(), embeddingForAll(EMBEDDINGS_MODEL, BATCH_INPUT)); + + assertNotNull(batch); + assertEquals(200, batch.getStatus()); + assertTrue(batch.getEntity() instanceof EmbeddingListView); + + final EmbeddingListView batchView = (EmbeddingListView) batch.getEntity(); + assertNotNull(batchView.usage()); + assertNotNull(batchView.usage().promptTokens()); + assertTrue("Usage must cover the whole batch, not one element of it", + batchView.usage().promptTokens() > singleView.usage().promptTokens()); + assertNotNull(batchView.usage().totalTokens()); + assertTrue(batchView.usage().totalTokens() > 0); + } + + /** + * Given one site whose {@code chat} section and {@code embeddings} section name different + * models + * When the same input is embedded twice, once naming the embeddings model and once naming the + * chat model + * Then the embeddings model is served and the chat model is refused as a model this site has + * not configured — because it is not configured for embeddings + * + *FR-011 and R7, stated as one assertion rather than two files apart. An implementation + * that validated against the chat section would fail exactly one of these two halves, and an + * implementation that validated against nothing at all would fail only the second — so both + * belong in the same test, against the same site, in the same configuration.
+ */ + @Test + public void test_embeddings_withSiteChatModel_isRefusedEvenThoughEmbeddingsModelSucceeds() { + final Response accepted = resource.embeddings( + mockRequest(host.getHostname(), bearerToken), mockResponse(), + host.getIdentifier(), embeddingFor(EMBEDDINGS_MODEL, INPUT_TEXT)); + + assertNotNull(accepted); + assertEquals("The model the site configured for embeddings must be served", + 200, accepted.getStatus()); + assertTrue(accepted.getEntity() instanceof EmbeddingListView); + + final Response refused = resource.embeddings( + mockRequest(host.getHostname(), bearerToken), mockResponse(), + host.getIdentifier(), embeddingFor(CHAT_MODEL, INPUT_TEXT)); + + assertNotNull(refused); + assertEquals("The site's chat model is not an embeddings model, however well configured " + + "it is for chat", 404, refused.getStatus()); + assertTrue(refused.getEntity() instanceof InferenceErrorView); + + final InferenceErrorView.Body error = ((InferenceErrorView) refused.getEntity()).error(); + assertNotNull(error); + assertEquals(ERROR_TYPE_INVALID_REQUEST, error.type()); + assertEquals("model", error.param()); + assertNotNull(error.message()); + assertTrue(error.message().contains(CHAT_MODEL)); + + // What the refusal must not leak: which model the site does embed with, where its provider + // lives, what key reaches it, or which site is behind the host name. + assertFalse(error.message().contains(EMBEDDINGS_MODEL)); + assertFalse(error.message().contains(AiTest.API_KEY)); + assertFalse(error.message().contains(String.valueOf(AiTest.PORT))); + assertFalse(error.message().contains(host.getHostname())); + assertFalse(error.message().contains(host.getIdentifier())); + + // Exactly one call reached the provider: the accepted one. The refusal never did. + wireMockServer.verify(1, postRequestedFor(urlPathEqualTo(EMBEDDINGS_PATH))); + } + + /** + * Given a model name the site has configured for no section at all + * When the embedding is requested + * Then it is refused with a 404 in the "no such model" shape and the provider is never + * contacted + */ + @Test + public void test_embeddings_withModelConfiguredForNothing_isRefusedAsNoSuchModel() { + final Response response = resource.embeddings( + mockRequest(host.getHostname(), bearerToken), mockResponse(), + host.getIdentifier(), embeddingFor(UNCONFIGURED_MODEL, INPUT_TEXT)); + + assertNotNull(response); + assertEquals(404, response.getStatus()); + assertTrue(response.getEntity() instanceof InferenceErrorView); + + final InferenceErrorView.Body error = ((InferenceErrorView) response.getEntity()).error(); + assertNotNull(error); + assertEquals(ERROR_TYPE_INVALID_REQUEST, error.type()); + assertEquals("model", error.param()); + assertNotNull(error.message()); + assertTrue(error.message().contains(UNCONFIGURED_MODEL)); + + wireMockServer.verify(0, postRequestedFor(urlPathEqualTo(EMBEDDINGS_PATH))); + } + + /** + * Given a request that omits {@code model} + * When the embedding is requested + * Then it is refused as a client error naming {@code model}, with no implicit default applied + * + *FR-024 holds across the family, not only on chat. A site has exactly one embeddings model + * configured, which makes defaulting to it look harmless — and is precisely why it has to be + * refused here: a caller who never named a model cannot tell when the site's changes.
+ */ + @Test + public void test_embeddings_withoutModel_isRejectedNamingTheField() { + final Response response = resource.embeddings( + mockRequest(host.getHostname(), bearerToken), mockResponse(), + host.getIdentifier(), embeddingFor(null, INPUT_TEXT)); + + assertNotNull(response); + assertEquals(400, response.getStatus()); + assertTrue(response.getEntity() instanceof InferenceErrorView); + + final InferenceErrorView.Body error = ((InferenceErrorView) response.getEntity()).error(); + assertNotNull(error); + assertEquals(ERROR_TYPE_INVALID_REQUEST, error.type()); + assertEquals("model", error.param()); + + wireMockServer.verify(0, postRequestedFor(urlPathEqualTo(EMBEDDINGS_PATH))); + } + + /** + * Given a request carrying no credential at all + * When the embedding is requested + * Then it is refused as unauthorized and the provider is never contacted + * + *FR-015. Accepts a thrown {@link WebApplicationException} as well as a returned 401, since + * the surrounding authentication handshake may refuse before the resource body runs and either + * is a valid refusal.
+ */ + @Test + public void test_embeddings_withNoCredential_isUnauthorized() { + final HttpServletRequest request = mockRequest(host.getHostname(), null); + + try { + final Response response = resource.embeddings( + request, mockResponse(), host.getIdentifier(), + embeddingFor(EMBEDDINGS_MODEL, INPUT_TEXT)); + assertEquals("An anonymous request must be refused as unauthorized", + Response.Status.UNAUTHORIZED.getStatusCode(), response.getStatus()); + assertTrue("a refusal carries the standard error shape", + response.getEntity() instanceof InferenceErrorView); + } catch (final WebApplicationException e) { + assertEquals(Response.Status.UNAUTHORIZED.getStatusCode(), + e.getResponse().getStatus()); + } + + wireMockServer.verify(0, postRequestedFor(urlPathEqualTo(EMBEDDINGS_PATH))); + } + + /** + * Given an authenticated request naming a site explicitly + * When the embedding is requested + * Then the resolved site is published on the request for the response filter to report + * + *FR-020 asks for the serving site on every response, and it reaches a real caller as the + * {@code X-dotCMS-Resolved-Site} header that {@link ResolvedSiteHeaderFilter} writes from + * {@link InferenceRequestAttributes#RESOLVED_SITE_ID}. This test calls the resource method + * directly, so no JAX-RS response filter runs and no header exists to read; the attribute is + * the filter's sole input and the only part of the chain this resource is responsible for.
+ */ + @Test + public void test_embeddings_withResolvedSite_publishesResolvedSiteId() { + final HttpServletRequest request = mockRequest(host.getHostname(), bearerToken); + + resource.embeddings(request, mockResponse(), host.getIdentifier(), + embeddingFor(EMBEDDINGS_MODEL, INPUT_TEXT)); + + assertEquals("The site whose configuration served the embedding must be published for the " + + "response filter to report", + host.getIdentifier(), + request.getAttribute(InferenceRequestAttributes.RESOLVED_SITE_ID)); + } + + /** + * Stubs the OpenAI-compatible provider: one canned vector for a single input, three for a + * batch, so that the response really does carry one entry per input rather than a fixed list + * the endpoint could return no matter what it was asked. + * + *The two stubs are separated by priority and by a body match on {@link #BATCH_MARKER}, the + * last element of {@link #BATCH_INPUT}, which can only appear in the provider request when a + * batch was sent. Both are registered above the default priority on purpose: WireMock also + * loads the checked-in mappings under {@code src/test/resources/mappings}, one of which answers + * any {@code POST /embeddings} carrying the shared test API key, and without explicit + * priorities this file would be asserting against whichever stub happened to win.
+ */ + private static void stubProvider() { + wireMockServer.stubFor(post(urlPathEqualTo(EMBEDDINGS_PATH)) + .atPriority(1) + .withRequestBody(containing(BATCH_MARKER)) + .willReturn(aResponse() + .withStatus(200) + .withHeader("Content-Type", "application/json") + .withBody(PROVIDER_BATCH_RESPONSE))); + + wireMockServer.stubFor(post(urlPathEqualTo(EMBEDDINGS_PATH)) + .atPriority(2) + .willReturn(aResponse() + .withStatus(200) + .withHeader("Content-Type", "application/json") + .withBody(PROVIDER_RESPONSE))); + } + + /** + * @param view the response to search + * @param index the index a caller sent an input at + * @return the entry that claims that index, or null when nothing does + */ + private static EmbeddingListView.EmbeddingView embeddingAtIndex(final EmbeddingListView view, + final int index) { + return view.data().stream() + .filter(embedding -> embedding.index() == index) + .findFirst() + .orElse(null); + } + + /** + * Asserts that an {@code input} FR-011 refuses is refused here — as a 400 in the standard + * error shape, naming the field, with nothing reaching the provider. + * + *Shared by the refusals that differ only in what was sent, so that a new one is a line + * rather than a copied block, and so that all of them stay pinned to the same param name: a + * caller cannot correct a payload they are not told the offending field of.
+ * + * @param input the {@code input} value to send, or null to omit the field entirely + * @return the error body, so that a caller with more to assert about the message can carry on + * from here rather than repeat all of this + */ + private InferenceErrorView.Body assertInputRefusedNamingTheField(final JsonNode input) { + final Response response = resource.embeddings( + mockRequest(host.getHostname(), bearerToken), mockResponse(), + host.getIdentifier(), new EmbeddingsRequestView(EMBEDDINGS_MODEL, input)); + + assertNotNull(response); + assertEquals("'" + input + "' is not something to embed", 400, response.getStatus()); + assertTrue(response.getEntity() instanceof InferenceErrorView); + + final InferenceErrorView.Body error = ((InferenceErrorView) response.getEntity()).error(); + assertNotNull(error); + assertEquals(ERROR_TYPE_INVALID_REQUEST, error.type()); + assertEquals("input", error.param()); + assertNotNull(error.message()); + assertFalse(error.message().isBlank()); + + wireMockServer.verify(0, postRequestedFor(urlPathEqualTo(EMBEDDINGS_PATH))); + + return error; + } + + /** + * Asserts everything {@link #assertInputRefusedNamingTheField(JsonNode)} does, and additionally + * that the message identifies which element of the array is at fault. + * + *FR-011 keeps {@code param} at {@code input} — that is the field the caller sent — so the + * message is the only place the offending position can appear, and a caller who batched five + * hundred strings should not have to bisect their own payload to find it. The index is the + * 0-based one the response's own {@code index} correlates on, so that the number in the error + * means the same thing as the number in a success.
+ * + *The match is on the index as a standalone number rather than on a substring, so that the + * digits of some unrelated value — a count, a limit, an element the message quotes — cannot + * stand in for the position. The tests calling this place the offending element at an index no + * other number in the payload shares, for the same reason.
+ * + * @param input the {@code input} array to send + * @param index the 0-based position of the element that is at fault + */ + private void assertInputRefusedIdentifyingTheElement(final JsonNode input, final int index) { + final InferenceErrorView.Body error = assertInputRefusedNamingTheField(input); + + assertTrue("A caller who sent a batch must be told which element is the problem, not just " + + "that 'input' is: the message must identify position " + index + + " (0-based, as the response's own index is), and '" + error.message() + + "' does not", + Pattern.compile("\\b" + index + "\\b").matcher(error.message()).find()); + } + + /** + * The scalar form of {@code input}: a single string, as + * {@code {"model": "…", "input": "The quick brown fox"}}. + * + * @param model the model to ask for, or null to omit the field + * @param input the text to embed + * @return the smallest well-formed embeddings request + */ + private static EmbeddingsRequestView embeddingFor(final String model, final String input) { + return new EmbeddingsRequestView(model, TextNode.valueOf(input)); + } + + /** + * The array form of {@code input}, as + * {@code {"model": "…", "input": ["The quick brown fox", "…"]}}. Called with no inputs it + * builds the empty array, which FR-011 refuses. + * + * @param model the model to ask for, or null to omit the field + * @param inputs the texts to embed, in the order the caller sent them + * @return an embeddings request carrying a batch + */ + private static EmbeddingsRequestView embeddingForAll(final String model, + final String... inputs) { + final ArrayNode input = OBJECT_MAPPER.createArrayNode(); + for (final String text : inputs) { + input.add(text); + } + + return new EmbeddingsRequestView(model, input); + } + + /** + * Builds a request arriving at a given host name, with or without a bearer credential. + * + *Request attributes are backed by a real map rather than left as mock no-ops, because both + * the authentication handshake and FR-020's site attribution publish through them, and a mock + * that forgot what was set on it would not behave like a servlet container.
+ * + * @param serverName the host name the request arrives on + * @param credential the {@code Authorization} header value, or null for no credential + * @return the mocked request + */ + private static HttpServletRequest mockRequest(final String serverName, + final String credential) { + final HttpServletRequest request = mock(HttpServletRequest.class); + final MapThe source issue asked for both fallbacks to be removed for this endpoint family. That was + * declined, so they need tests that state the kept behaviour rather than tests that would pass + * either way: a family which resolved differently from the rest of dotCMS would break the + * server-side callers it exists for, because internal DNS names, container service names and + * {@code localhost} are not site aliases, and because many installations configure dotAI once at + * system level rather than per site.
+ * + *The provider is a WireMock server standing in for an OpenAI-compatible endpoint, wired in + * through the same dotAI app secrets the rest of the AI integration tests use, so the exchange + * travels the real client path rather than a stubbed one.
+ * + *These tests move the system-level dotAI configuration, which is shared + * state. Whatever SYSTEM_HOST carried when the class started is captured and put back in + * {@link #afterClass()}.
+ */ +public class InferenceFallbackTest { + + /** The chat model the system-level configuration carries, and the only one asked for here. */ + private static final String CHAT_MODEL = "gpt-4o-mini"; + + /** Path an OpenAI-compatible provider serves completions on. */ + private static final String COMPLETIONS_PATH = "/chat/completions"; + + /** + * A host name no site and no alias can match, standing in for the internal DNS name, container + * service name or {@code localhost} a server-side caller routinely presents. + */ + private static final String UNMATCHED_HOST_NAME = "inference-fallback-no-such-site.invalid"; + + /** What the stubbed provider answers whenever it is reached at all. */ + private static final String PROVIDER_RESPONSE = """ + { + "id": "chatcmpl-fallback-1", + "object": "chat.completion", + "created": 1789000000, + "model": "gpt-4o-mini", + "choices": [ + { + "index": 0, + "message": { + "role": "assistant", + "content": "Served." + }, + "finish_reason": "stop" + } + ], + "usage": {"prompt_tokens": 9, "completion_tokens": 2, "total_tokens": 11} + } + """; + + private static WireMockServer wireMockServer; + private static User user; + private static String bearerToken; + private static Host defaultHost; + + /** The system-level providerConfig as found, so these tests can put it back. */ + private static String originalSystemProviderConfig; + + private Host unconfiguredSite; + private final ChatCompletionsResource resource = new ChatCompletionsResource(); + + @BeforeClass + public static void beforeClass() throws Exception { + IntegrationTestInitService.getInstance().init(); + IPUtils.disabledIpPrivateSubnet(true); + wireMockServer = AiTest.prepareWireMock(); + stubProvider(); + + defaultHost = APILocator.getHostAPI().findDefaultHost(APILocator.systemUser(), false); + originalSystemProviderConfig = + ConfigService.INSTANCE.config(APILocator.systemHost()).getProviderConfig(); + + // Both roles are needed, not one: WebResource.checkRolePermissions matches + // DOTCMS_BACK_END_USER by key without walking inheritance, so being an admin does not + // imply it, and admin is what grants read on the site passed as an explicit override. + user = new UserDataGen() + .roles(APILocator.getRoleAPI().loadBackEndUserRole(), + APILocator.getRoleAPI().loadCMSAdminRole()) + .nextPersisted(); + final ApiToken apiToken = APILocator.getApiTokenAPI().persistApiToken( + user.getUserId(), + Date.from(Instant.now().plus(Duration.ofDays(1))), + APILocator.systemUser().getUserId(), + "127.0.0.1"); + bearerToken = "Bearer " + APILocator.getApiTokenAPI().getJWT(apiToken, user); + } + + @AfterClass + public static void afterClass() throws Exception { + restoreSystemLevelConfiguration(); + wireMockServer.stop(); + IPUtils.disabledIpPrivateSubnet(false); + } + + @Before + public void before() { + unconfiguredSite = new SiteDataGen().nextPersisted(); + wireMockServer.resetRequests(); + } + + @After + public void after() throws Exception { + clearSystemLevelConfiguration(); + } + + /** + * Given a request whose host name matches no site and no alias, and a system-level dotAI + * configuration the default site inherits + * When the completion is requested with no explicit site override + * Then it is served rather than refused + */ + @Test + public void test_completions_withUnmatchedHostName_isServedByDefaultSite() throws Exception { + configureSystemLevel(); + final HttpServletRequest request = mockRequest(UNMATCHED_HOST_NAME); + + final Response response = resource.completions( + request, mockResponse(), null, completionFor(CHAT_MODEL)); + + assertNotNull(response); + assertEquals(200, response.getStatus()); + assertTrue(response.getEntity() instanceof ChatCompletionView); + + final ChatCompletionView view = (ChatCompletionView) response.getEntity(); + assertNotNull(view.choices()); + assertEquals(1, view.choices().size()); + assertNotNull(view.choices().get(0).message()); + assertNotNull(view.choices().get(0).message().content()); + assertFalse(view.choices().get(0).message().content().isBlank()); + } + + /** + * Given a request whose host name matches no site and no alias + * When the completion is requested with no explicit site override + * Then the site reported as having served is the default site + * + *FR-020 asks for the fallback to be logged as well. Asserting on the log would mean + * attaching an appender to dotCMS's Log4j configuration from an integration test, which is + * both brittle and outside what this family owns, so what is asserted here is the observable + * half of the same requirement: the serving site published for the response header. The log + * line itself lives in {@code AiHostResolver.resolveFromRequest} and carries the unmatched + * host name.
+ */ + @Test + public void test_completions_withUnmatchedHostName_reportsDefaultSiteAsServingSite() + throws Exception { + configureSystemLevel(); + final HttpServletRequest request = mockRequest(UNMATCHED_HOST_NAME); + + resource.completions(request, mockResponse(), null, completionFor(CHAT_MODEL)); + + verify(request).setAttribute( + InferenceRequestAttributes.RESOLVED_SITE_ID, defaultHost.getIdentifier()); + } + + /** + * Given a site with no dotAI configuration of its own and a system-level configuration + * When the completion is requested against that site + * Then it is served from the system-level configuration and the serving site is still the + * requested one + */ + @Test + public void test_completions_siteWithoutOwnConfiguration_isServedFromSystemLevel() + throws Exception { + configureSystemLevel(); + final HttpServletRequest request = mockRequest(UNMATCHED_HOST_NAME); + + final Response response = resource.completions( + request, mockResponse(), unconfiguredSite.getIdentifier(), completionFor(CHAT_MODEL)); + + assertNotNull(response); + assertEquals(200, response.getStatus()); + assertTrue(response.getEntity() instanceof ChatCompletionView); + + // The configuration was inherited, but the site that resolved — and that a reconciliation + // pipeline would bill — is the one the caller named, not SYSTEM_HOST. + verify(request).setAttribute( + InferenceRequestAttributes.RESOLVED_SITE_ID, unconfiguredSite.getIdentifier()); + wireMockServer.verify(1, postRequestedFor(urlPathEqualTo(COMPLETIONS_PATH))); + } + + /** + * Given a site with no dotAI configuration of its own and no system-level configuration either + * When the completion is requested against that site + * Then it is refused with an explanatory error and no provider is contacted + * + *The refusal is the 404 "no such model" shape rather than a distinct "site not + * configured" error, deliberately: from where the caller stands a model nobody configured and + * a model on a site nobody configured are the same absence, and separating them would tell a + * caller which sites have dotAI set up.
+ */ + @Test + public void test_completions_withNoConfigurationAnywhere_isRefused() throws Exception { + clearSystemLevelConfiguration(); + final HttpServletRequest request = mockRequest(UNMATCHED_HOST_NAME); + + final Response response = resource.completions( + request, mockResponse(), unconfiguredSite.getIdentifier(), completionFor(CHAT_MODEL)); + + assertNotNull(response); + assertEquals(404, response.getStatus()); + assertTrue(response.getEntity() instanceof InferenceErrorView); + + final InferenceErrorView errorView = (InferenceErrorView) response.getEntity(); + assertNotNull(errorView.error()); + assertNotNull(errorView.error().message()); + assertFalse(errorView.error().message().isBlank()); + assertEquals("model", errorView.error().param()); + + // Nothing was served from anywhere: no unrelated site's credentials were spent. + wireMockServer.verify(0, postRequestedFor(urlPathEqualTo(COMPLETIONS_PATH))); + } + + /** + * Installs the dotAI configuration at system level, where every site inherits it from. + */ + private static void configureSystemLevel() throws Exception { + AiTest.aiAppSecretsWithProviderConfig( + APILocator.systemHost(), AiTest.providerConfigJson(AiTest.PORT, CHAT_MODEL)); + } + + /** + * Removes the system-level dotAI configuration and waits until it has stopped resolving, so a + * test asserting the unconfigured case is not racing the secrets cache. + */ + private static void clearSystemLevelConfiguration() throws Exception { + AiTest.removeAiAppSecrets(APILocator.systemHost()); + await().atMost(5, SECONDS) + .until(() -> !ConfigService.INSTANCE.config(APILocator.systemHost()).isEnabled()); + } + + /** + * Puts SYSTEM_HOST back the way this class found it. The system-level dotAI configuration is + * shared with every other test in the JVM, so leaving it moved would be a defect in this file + * rather than in the code under test. + */ + private static void restoreSystemLevelConfiguration() throws Exception { + if (originalSystemProviderConfig != null && !originalSystemProviderConfig.isBlank()) { + AiTest.aiAppSecretsWithProviderConfig( + APILocator.systemHost(), originalSystemProviderConfig); + } else { + clearSystemLevelConfiguration(); + } + } + + /** + * Stubs the OpenAI-compatible provider with a single canned answer; which model or site + * reached it is not what this file is about, only whether it was reached at all. + */ + private static void stubProvider() { + wireMockServer.stubFor(post(urlPathEqualTo(COMPLETIONS_PATH)) + .willReturn(aResponse() + .withStatus(200) + .withHeader("Content-Type", "application/json") + .withBody(PROVIDER_RESPONSE))); + } + + /** + * @param model the model to ask for + * @return the smallest valid completion request + */ + private static ChatCompletionRequestView completionFor(final String model) { + return new ChatCompletionRequestView( + model, + List.of(new MessageView("user", "Say something.", null, null, null)), + null, null, null, null, null, null, null, null, null, null); + } + + /** + * @param serverName the host name the caller presented + * @return a request authenticated with the test user's bearer token, as the family requires + */ + private static HttpServletRequest mockRequest(final String serverName) { + final HttpServletRequest request = mock(HttpServletRequest.class); + when(request.getRequestURI()).thenReturn("/api/inference/v1/chat/completions"); + when(request.getRequestURL()) + .thenReturn(new StringBuffer("http://localhost/api/inference/v1/chat/completions")); + when(request.getMethod()).thenReturn("POST"); + when(request.getServerName()).thenReturn(serverName); + when(request.getRemoteAddr()).thenReturn("127.0.0.1"); + when(request.getHeader("Authorization")).thenReturn(bearerToken); + return request; + } + + private static HttpServletResponse mockResponse() { + return mock(HttpServletResponse.class); + } +} diff --git a/dotcms-integration/src/test/java/com/dotcms/inference/rest/InferenceImagesTest.java b/dotcms-integration/src/test/java/com/dotcms/inference/rest/InferenceImagesTest.java new file mode 100644 index 000000000000..aedf5803ab29 --- /dev/null +++ b/dotcms-integration/src/test/java/com/dotcms/inference/rest/InferenceImagesTest.java @@ -0,0 +1,968 @@ +package com.dotcms.inference.rest; + +import com.dotcms.ai.AiTest; +import com.dotcms.auth.providers.jwt.beans.ApiToken; +import com.dotcms.datagen.SiteDataGen; +import com.dotcms.datagen.UserDataGen; +import com.dotcms.inference.model.InferenceLimits; +import com.dotcms.inference.rest.view.ImageGenerationRequestView; +import com.dotcms.inference.rest.view.ImageGenerationView; +import com.dotcms.inference.rest.view.InferenceErrorView; +import com.dotcms.util.IntegrationTestInitService; +import com.dotcms.util.network.IPUtils; +import com.dotmarketing.beans.Host; +import com.dotmarketing.business.APILocator; +import com.fasterxml.jackson.databind.JsonNode; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.github.tomakehurst.wiremock.WireMockServer; +import com.liferay.portal.model.User; +import org.junit.After; +import org.junit.AfterClass; +import org.junit.Before; +import org.junit.BeforeClass; +import org.junit.Test; + +import javax.servlet.http.HttpServletRequest; +import javax.servlet.http.HttpServletResponse; +import javax.ws.rs.WebApplicationException; +import javax.ws.rs.core.Response; +import java.time.Duration; +import java.time.Instant; +import java.util.Base64; +import java.util.Date; +import java.util.HashMap; +import java.util.Map; +import java.util.UUID; + +import static com.github.tomakehurst.wiremock.client.WireMock.aResponse; +import static com.github.tomakehurst.wiremock.client.WireMock.containing; +import static com.github.tomakehurst.wiremock.client.WireMock.matchingJsonPath; +import static com.github.tomakehurst.wiremock.client.WireMock.post; +import static com.github.tomakehurst.wiremock.client.WireMock.postRequestedFor; +import static com.github.tomakehurst.wiremock.client.WireMock.urlPathEqualTo; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.anyString; +import static org.mockito.Mockito.doAnswer; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; + +/** + * Specifies {@code POST /api/inference/v1/images/generations} through + * {@link ImagesResource#generations(HttpServletRequest, HttpServletResponse, String, ImageGenerationRequestView)}. + * + *Two decisions in FR-012 are the reason this file exists rather than a copy of the chat tests + * with a different noun.
+ * + *The image comes back inline, as base64, and never as a URL. That is a + * deliberate choice and not an incidental one: a hosted URL would mean deciding storage, + * authentication and lifetime for an artifact generated from a prompt that may carry customer + * data, so the family avoids creating a separately-addressable artifact at all. A decision like + * that reverses quietly — an implementation that passes the provider's {@code url} straight + * through returns a 200 with a perfectly plausible body — so it is asserted on the serialized wire + * shape, where a {@code url} property appearing later is caught whether or not anyone remembers + * why it must not.
+ * + *Deliberately not asserted here: that dotCMS asks the provider for base64 upstream. + * FR-012 does require it — a provider-minted URL is a real artifact on someone else's + * infrastructure, frequently public and long-lived, which is a larger exposure than anything this + * family's own response shape can create — but it requires it only "where a provider offers the + * choice", and the seven configured vendors do not agree on whether that choice exists or what it + * is called. Where no such choice exists the requirement says the opposite thing: dotCMS fetches + * the provider's URL and re-encodes it rather than refusing the provider, and FR-012 now states + * the residual openly — on those providers an addressable artifact really does exist upstream, and + * this family's guarantee is only that a caller never receives one. An assertion on the outbound + * request body would therefore pin one vendor's spelling of a field the requirement itself makes + * conditional, and would fail the moment the provider abstraction changed how it phrases a request + * it is still phrasing correctly. It is the same reason the streaming tests stopped asserting that + * {@code include_usage} never reached the provider: what is on the wire upstream belongs to the + * client library, and testing it tests the library. What this family owns, and what is asserted + * below, is that the response a caller receives carries {@code b64_json} and no + * {@code url}. This paragraph exists so that the absence reads as a decision rather than as an + * oversight somebody helpfully corrects.
+ * + *{@code size}, by contrast, is asserted on the outbound request, and the + * difference is not inconsistency. Asking for the inline form is a provider capability — whether + * the ask exists at all varies by vendor, so a test of it is a test of the vendor. Passing a + * caller's {@code size} through is dotCMS's own behaviour: the caller named a size, FR-012 says it + * reaches the provider "where the provider accepts it", and the stub here is a provider that + * accepts it. It is the same logic FR-013 applies to the sampling parameters — {@code size} + * changes both what the caller receives and what the site pays for, so dropping it silently is a + * cost and correctness failure rather than a compatibility courtesy, and a dropped {@code size} is + * invisible in the response: a 1024x1024 image is a perfectly plausible answer to a request for + * something else.
+ * + *The model is validated against the site's {@code image} section, not its + * chat models and not its embeddings model. A site configures all three separately, so reusing the + * chat gate here would accept the wrong model and refuse the right one while still answering + * 200/404 in the right shapes.
+ * + *The provider is a WireMock server standing in for an OpenAI-compatible endpoint, wired in + * through the same dotAI app secrets the rest of the AI integration tests use, so an accepted call + * travels the real client path rather than a stubbed one.
+ */ +public class InferenceImagesTest { + + /** + * The model the site's {@code image} section is configured with. Read from {@link AiTest} + * rather than restated, because the three-way separation this file relies on is a property of + * that shared configuration. + */ + private static final String IMAGE_MODEL = AiTest.IMAGE_MODEL; + + /** The model the same site's {@code embeddings} section is configured with. */ + private static final String EMBEDDINGS_MODEL = AiTest.EMBEDDINGS_MODEL; + + /** The model the same site's {@code chat} section is configured with. */ + private static final String CHAT_MODEL = "gpt-4o-mini"; + + /** Path an OpenAI-compatible provider serves image generation on. */ + private static final String IMAGES_PATH = "/images/generations"; + + private static final String REQUEST_URI = "/api/inference/v1/images/generations"; + + private static final String ERROR_TYPE_INVALID_REQUEST = "invalid_request_error"; + + /** The prompt these tests generate from. */ + private static final String PROMPT = "A cat in a hammock"; + + /** The size these tests ask for; the one every configured provider supports. */ + private static final String SIZE = AiTest.IMAGE_SIZE; + + /** + * A second size the image model supports, asked for only by the pass-through test. + * + *It has to differ from {@link #SIZE} for that test to mean anything: a request that asks + * for the default cannot distinguish a size that travelled from a size the provider would have + * used anyway. Neither string is a substring of the other, so the body match that separates the + * two stubs cannot fire on the wrong request.
+ */ + private static final String ALTERNATE_SIZE = "1792x1024"; + + /** A 1x1 PNG. Small enough to read, and real base64, so decoding it proves something. */ + private static final String IMAGE_BASE64 = + "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8z8BQDwAEhQGAhKmMIQ" + + "AAAABJRU5ErkJggg=="; + + /** + * A different 1x1 PNG — a second real image, distinct from {@link #IMAGE_BASE64} — served only + * by the stub that matches on {@link #ALTERNATE_SIZE}. It is what makes "the size reached the + * provider" visible in the response as well as in the request log: an implementation that drops + * {@code size} falls through to the catch-all stub and comes back with the other image. + */ + private static final String ALTERNATE_IMAGE_BASE64 = + "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNk+M9QDwADhgGAWjR9aw" + + "AAAABJRU5ErkJggg=="; + + /** How many images the multi-image test asks for: the smallest number that is not one. */ + private static final int TWO_IMAGES = 2; + + /** An image model on a provider that implements only the single-image call. */ + private static final String GEMINI_IMAGE_MODEL = "gemini-2.5-flash-image"; + + /** + * What a provider request that really asked for {@link #TWO_IMAGES} images carries, and the + * body match that separates the two-image stub from the single-image one. + * + *It has to be the outbound {@code n} rather than anything about the prompt: a stub keyed on + * the prompt would answer two images to an implementation that never told the provider how many + * it wanted, and the test would pass on a response the provider was never asked for.
+ * + *Matched as a JSON path rather than as a substring of the body. The provider client writes + * its request pretty-printed — {@code "n" : 2}, with spaces around the colon — so a substring + * match on {@code "n":2} silently fails to match a request that did carry the field, falls + * through to the single-image stub, and reports the implementation as returning one image when + * what actually went wrong was the assertion. A path match is indifferent to formatting.
+ */ + private static final String TWO_IMAGES_MARKER = "$[?(@.n == " + TWO_IMAGES + ")]"; + + /** + * A third 1x1 PNG, distinct from both {@link #IMAGE_BASE64} and {@link #ALTERNATE_IMAGE_BASE64}, + * carried by the second entry of {@link #PROVIDER_TWO_IMAGES_RESPONSE}. + * + *The second entry has to be a different image from the first for the multi-image test to + * mean anything: two copies of one payload would satisfy "two entries in {@code data}" while + * the implementation generated once and duplicated the answer. It is the same hole the batch + * embeddings test closes by asserting that its vectors are distinct.
+ */ + private static final String SECOND_IMAGE_BASE64 = + "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAIAAACQd1PeAAAADElEQVR42mNg+M8AAAICAQBF9FLUAA" + + "AAAElFTkSuQmCC"; + + /** + * What the stubbed provider answers. Modelled on the checked-in dotAI image stubs under + * {@code src/test/resources/mappings} — {@code created}, and a {@code data} entry carrying the + * provider's rewritten prompt — except that the payload is the image itself rather than a + * hosted link, which is what an OpenAI-compatible provider returns for a base64 request and + * what FR-012 requires this family to deal in. + */ + private static final String PROVIDER_RESPONSE = """ + { + "created": 1789000000, + "data": [ + { + "revised_prompt": "A contented cat asleep in a woven hammock, warm afternoon light.", + "b64_json": "%s" + } + ] + } + """.formatted(IMAGE_BASE64); + + /** + * What the stubbed provider answers a request that carried {@link #ALTERNATE_SIZE}: the same + * shape, a different image. Only a request whose body really contains the size the caller asked + * for can reach this stub — see {@link #stubProvider()}. + */ + private static final String PROVIDER_ALTERNATE_SIZE_RESPONSE = """ + { + "created": 1789000000, + "data": [ + { + "revised_prompt": "A contented cat asleep in a wide woven hammock, warm afternoon light.", + "b64_json": "%s" + } + ] + } + """.formatted(ALTERNATE_IMAGE_BASE64); + + /** + * What the stubbed provider answers a request that really asked for {@link #TWO_IMAGES} + * images: two entries, each carrying its own image, which is what an OpenAI-compatible + * provider returns for an {@code n} of two. Only a request whose body carries + * {@link #TWO_IMAGES_MARKER} can reach this stub — see {@link #stubProvider()}. + */ + private static final String PROVIDER_TWO_IMAGES_RESPONSE = """ + { + "created": 1789000000, + "data": [ + { + "revised_prompt": "A contented cat asleep in a woven hammock, warm afternoon light.", + "b64_json": "%s" + }, + { + "revised_prompt": "A tabby cat curled in a rope hammock, late afternoon sun.", + "b64_json": "%s" + } + ] + } + """.formatted(IMAGE_BASE64, SECOND_IMAGE_BASE64); + + private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper(); + + private static WireMockServer wireMockServer; + + /** A backend user who is also an administrator, so site READ is never the thing under test. */ + private static User user; + private static String bearerToken; + + private Host host; + private final ImagesResource resource = new ImagesResource(); + + @BeforeClass + public static void beforeClass() throws Exception { + IntegrationTestInitService.getInstance().init(); + IPUtils.disabledIpPrivateSubnet(true); + wireMockServer = AiTest.prepareWireMock(); + stubProvider(); + + // A bare UserDataGen user has no roles at all, so it is neither a backend nor a frontend + // user and FR-016 rejects it; it also cannot read the site these tests pass as an explicit + // override, which FR-019 checks. Two roles are needed, not one: the check in + // WebResource.checkRolePermissions is doesUserHaveRole(user, "DOTCMS_BACK_END_USER") by key + // and does not walk inheritance, so being an admin does not imply it. Admin is what grants + // read on the site. Role-specific behaviour is US3's tests, not these. + user = new UserDataGen() + .roles(APILocator.getRoleAPI().loadBackEndUserRole(), + APILocator.getRoleAPI().loadCMSAdminRole()) + .nextPersisted(); + final ApiToken apiToken = APILocator.getApiTokenAPI().persistApiToken( + user.getUserId(), + Date.from(Instant.now().plus(Duration.ofDays(1))), + APILocator.systemUser().getUserId(), + "127.0.0.1"); + bearerToken = "Bearer " + APILocator.getApiTokenAPI().getJWT(apiToken, user); + } + + @AfterClass + public static void afterClass() { + wireMockServer.stop(); + IPUtils.disabledIpPrivateSubnet(false); + } + + @Before + public void before() throws Exception { + host = new SiteDataGen() + .name("inference-images-" + UUID.randomUUID() + ".dotcms.com") + .nextPersisted(); + // Configures chat, embeddings and image as three separate sections, each with its own + // model. The chat and embeddings models given here are the ones this file expects the + // image gate to refuse. + AiTest.aiAppSecretsWithProviderConfig( + host, AiTest.providerConfigJson(AiTest.PORT, CHAT_MODEL)); + wireMockServer.resetRequests(); + } + + @After + public void after() throws Exception { + AiTest.removeAiAppSecrets(host); + } + + /** + * Given a site configured with an image model, and a request carrying a prompt + * When the image is requested + * Then it comes back in the standard image-generation shape — a {@code created} timestamp and + * one entry in {@code data} + */ + @Test + public void test_generations_withPrompt_returnsImageInStandardShape() { + final Response response = resource.generations( + mockRequest(host.getHostname(), bearerToken), mockResponse(), + host.getIdentifier(), generationFor(IMAGE_MODEL, PROMPT)); + + assertNotNull(response); + assertEquals(200, response.getStatus()); + assertTrue(response.getEntity() instanceof ImageGenerationView); + + final ImageGenerationView view = (ImageGenerationView) response.getEntity(); + assertTrue("The answer must carry a creation timestamp", view.created() > 0); + assertNotNull(view.data()); + assertEquals(1, view.data().size()); + assertNotNull(view.data().get(0)); + + wireMockServer.verify(1, postRequestedFor(urlPathEqualTo(IMAGES_PATH))); + } + + /** + * Given a generated image + * When the answer is serialized as a client would receive it + * Then the entry carries a non-blank, decodable {@code b64_json} payload and no hosted + * {@code url} + * + *FR-012. The assertion is made against the serialized wire shape rather than against the + * view's accessors, because that is where the decision can be undone: a {@code url} property + * added to the payload later — by widening the view, or by letting the provider's own entry + * through unmapped — would restore exactly the separately-addressable artifact the requirement + * refuses to create, while every status-code and shape assertion in this file kept passing. + * Decoding the payload is what separates a real image from a placeholder string that merely + * occupies the field.
+ */ + @Test + public void test_generations_withGeneratedImage_returnsBase64AndNoHostedUrl() { + final Response response = resource.generations( + mockRequest(host.getHostname(), bearerToken), mockResponse(), + host.getIdentifier(), generationFor(IMAGE_MODEL, PROMPT)); + + assertNotNull(response); + assertEquals(200, response.getStatus()); + assertTrue(response.getEntity() instanceof ImageGenerationView); + + final ImageGenerationView view = (ImageGenerationView) response.getEntity(); + final String b64Json = view.data().get(0).b64Json(); + assertNotNull("FR-012 returns the image inline, so b64_json is not optional", b64Json); + assertFalse(b64Json.isBlank()); + + try { + assertTrue("An empty payload is not an image", + Base64.getDecoder().decode(b64Json).length > 0); + } catch (final IllegalArgumentException e) { + fail("b64_json must be valid base64, not an opaque placeholder: " + e.getMessage()); + } + + final JsonNode payload = OBJECT_MAPPER.valueToTree(view); + final JsonNode entry = payload.get("data").get(0); + assertTrue("The wire payload names the image b64_json, as standard clients read it", + entry.hasNonNull("b64_json")); + assertEquals(b64Json, entry.get("b64_json").asText()); + assertFalse("FR-012 creates no hosted, separately-addressable artifact, so no url may " + + "reach the caller", entry.hasNonNull("url")); + } + + /** + * Given requests asking for no images and for a negative number of them + * When each generation is requested + * Then each is refused as a client error naming {@code n}, and the provider is never contacted + * + *FR-012 draws its line below 1, not around it: zero and a negative are not quantities of + * images a caller can mean, and they are the two ways of expressing that. They are also what a + * clamp swallows — {@code Math.max(1, n)} answers a request for no images with an image and a + * request for minus one with a bill, while every other assertion in this file keeps passing. + * Zero is not an empty success either: a caller who sends it has made a mistake, and answering + * it with an empty {@code data} list would be the same false success FR-011 refuses for an + * empty batch of embeddings.
+ * + *That is why the provider must not be contacted at all. Clamping is the plausible + * implementation of this field — the request reaches the provider once, a perfectly valid + * single-image 200 comes back, and every other assertion in this file still passes. A refusal + * that never leaves dotCMS is the only observable difference.
+ * + *An {@code n} of 2 is deliberately not among these values. An earlier draft of FR-012 + * refused any {@code n} other than 1, on the claim that the provider abstraction returns a + * single image per call; the claim was false — {@code ImageModel.generate(prompt, n)} returns a + * list, and the adopted format documents several images per request — and the requirement was + * corrected. Two images are served, by + * {@link #test_generations_withNOfTwo_returnsTwoDistinctImages()}.
+ */ + @Test + public void test_generations_withNBelowOne_isRejectedNamingTheField() { + assertCountRefusedNamingTheField(0); + assertCountRefusedNamingTheField(-1); + } + + /** + * Given a request asking for two images + * When the generation is requested + * Then two entries come back in {@code data}, each carrying its own non-blank {@code b64_json}, + * and the two are not the same image + * + *FR-012 as corrected: {@code n} is honored, because the adopted format supports several + * images in one request and this site's provider implements the multi-image call. Refusing would be + * this family declining something both the standard and this provider support, so the count a + * caller asks for is the count they receive. A provider that cannot is a different case, with + * its own test below.
+ * + *The two payloads are asserted to differ, not merely to be present. "Two entries" + * is a count an implementation can reach without generating twice — by asking the provider for + * one image and copying the answer into a list of the right length — and that implementation + * returns the caller one image under two headings while passing every shape assertion in this + * file. It is the same hole the batch embeddings test closes by asserting that its three + * vectors are distinct.
+ * + *The outbound request is verified as well, for the reason the {@code size} test gives: what + * the provider was asked for is invisible in the response. The stub that serves two images is + * keyed on that outbound {@code n}, so an implementation that never sends it falls through to + * the single-image stub and fails on the count too — and one that loops, calling the provider + * once per image, doubles the round trips and leaves a request half-charged when the second + * call fails after the first succeeded, which the single matching call catches.
+ */ + @Test + public void test_generations_withNOfTwo_returnsTwoDistinctImages() { + final Response response = resource.generations( + mockRequest(host.getHostname(), bearerToken), mockResponse(), + host.getIdentifier(), generationFor(IMAGE_MODEL, PROMPT, TWO_IMAGES)); + + assertNotNull(response); + assertEquals("n is honored, so a request for two images is served rather than refused", + 200, response.getStatus()); + assertTrue(response.getEntity() instanceof ImageGenerationView); + + final ImageGenerationView view = (ImageGenerationView) response.getEntity(); + assertTrue("The answer must carry a creation timestamp", view.created() > 0); + assertNotNull(view.data()); + assertEquals("A request for two images is two entries in data, not one", + TWO_IMAGES, view.data().size()); + + for (int index = 0; index < TWO_IMAGES; index++) { + final ImageGenerationView.ImageView entry = view.data().get(index); + assertNotNull("Entry " + index + " is missing entirely", entry); + assertNotNull("FR-012 returns every image inline, so entry " + index + + " carries a b64_json", entry.b64Json()); + assertFalse("A blank payload is not an image, and entry " + index + " carries one", + entry.b64Json().isBlank()); + } + + assertEquals("Each entry must carry a distinct image; two copies of one would mean a " + + "single image was generated and handed back twice", + TWO_IMAGES, + view.data().stream() + .map(ImageGenerationView.ImageView::b64Json) + .distinct().count()); + + // One request, asking for two images — not two requests asking for one each. + wireMockServer.verify(1, postRequestedFor(urlPathEqualTo(IMAGES_PATH)) + .withRequestBody(matchingJsonPath(TWO_IMAGES_MARKER))); + } + + /** + * Given a request naming a size other than the one the provider would default to + * When the image is requested + * Then the size reaches the provider, and the answer is the one only a provider that was told + * that size returns + * + *FR-012 requires {@code size} to be passed through where the provider accepts it, on the + * same reasoning FR-013 applies to the sampling parameters: it changes both what the caller + * receives and what the site pays, so dropping it is a cost and correctness failure rather than + * the harmless compatibility courtesy that ignoring an incidental field is. Unlike the base64 + * ask this file deliberately does not assert, passing a caller's own parameter through is + * dotCMS's behaviour rather than a provider capability that varies by vendor — the stub here is + * a provider that accepts {@code size}, and that is all FR-012 conditions the requirement + * on.
+ * + *It has to be asserted on the outbound request because it is invisible in the response: a + * provider asked for nothing in particular returns a perfectly well-formed image, and every + * shape assertion in this file passes while the caller receives a size they did not ask for and + * a bill for a size they did not choose. The stub matched on the size makes the two outcomes + * distinguishable twice over — the request log records a body carrying the size, and the answer + * carries the image only that stub serves, so an implementation that drops {@code size} falls + * through to the catch-all and fails on the payload as well as on the verification.
+ */ + @Test + public void test_generations_withSize_passesTheSizeToTheProvider() { + final Response response = resource.generations( + mockRequest(host.getHostname(), bearerToken), mockResponse(), + host.getIdentifier(), generationFor(IMAGE_MODEL, PROMPT, 1, ALTERNATE_SIZE)); + + assertNotNull(response); + assertEquals(200, response.getStatus()); + assertTrue(response.getEntity() instanceof ImageGenerationView); + + final ImageGenerationView view = (ImageGenerationView) response.getEntity(); + assertNotNull(view.data()); + assertEquals(1, view.data().size()); + assertEquals("The answer must be the one the provider gave for the size the caller asked " + + "for, not the one it gives when asked for nothing in particular", + ALTERNATE_IMAGE_BASE64, view.data().get(0).b64Json()); + + wireMockServer.verify(1, postRequestedFor(urlPathEqualTo(IMAGES_PATH)) + .withRequestBody(containing(ALTERNATE_SIZE))); + } + + /** + * Given two requests that differ only in {@code n} — one asking for exactly 1, one omitting + * the field + * When each image is generated + * Then both are served, each reaching the provider once and each answering with a single + * entry + * + *The boundary from the other side, and the reason it is a separate test rather than + * another assertion inside one: what FR-012 refuses is an {@code n} below 1, not the presence + * of the field, and not the absence of it either. An implementation that rejected any request + * carrying {@code n} would satisfy the refusal test on its own while breaking every standard + * client that sends the format's own default of 1, and an implementation that rejected a + * request omitting {@code n} would break the clients that leave it out. Both halves are + * asserted against the same site, in the same configuration, so neither can be read as a + * configuration accident.
+ */ + @Test + public void test_generations_withNOfOneOrAbsent_returnsOneImage() { + final Response explicitOne = resource.generations( + mockRequest(host.getHostname(), bearerToken), mockResponse(), + host.getIdentifier(), generationFor(IMAGE_MODEL, PROMPT, 1)); + + assertNotNull(explicitOne); + assertEquals("n of 1 is what a standard client sends by default, and one image is what " + + "it must answer with", 200, explicitOne.getStatus()); + assertTrue(explicitOne.getEntity() instanceof ImageGenerationView); + assertEquals(1, ((ImageGenerationView) explicitOne.getEntity()).data().size()); + + final Response absent = resource.generations( + mockRequest(host.getHostname(), bearerToken), mockResponse(), + host.getIdentifier(), generationFor(IMAGE_MODEL, PROMPT, null)); + + assertNotNull(absent); + assertEquals("n is optional; omitting it means one image, not a malformed request", + 200, absent.getStatus()); + assertTrue(absent.getEntity() instanceof ImageGenerationView); + assertEquals(1, ((ImageGenerationView) absent.getEntity()).data().size()); + + wireMockServer.verify(2, postRequestedFor(urlPathEqualTo(IMAGES_PATH))); + } + + /** + * Given one site whose {@code chat}, {@code embeddings} and {@code image} sections name three + * different models + * When the same prompt is generated from three times, once naming each model + * Then only the image model is served; the chat model and the embeddings model are both + * refused as models this site has not configured — because neither is configured + * for images + * + *The three halves belong in one test, against one site, in one configuration. An + * implementation validating against the chat section would fail the first refusal; one + * validating against the embeddings section would fail the second; one validating against + * nothing would fail both while still serving the accepted case. Split across three tests, any + * single one of them could be read as flaky configuration rather than as the wrong section.
+ */ + @Test + public void test_generations_withSiteChatOrEmbeddingsModel_isRefusedEvenThoughImageModelSucceeds() { + final Response accepted = resource.generations( + mockRequest(host.getHostname(), bearerToken), mockResponse(), + host.getIdentifier(), generationFor(IMAGE_MODEL, PROMPT)); + + assertNotNull(accepted); + assertEquals("The model the site configured for images must be served", + 200, accepted.getStatus()); + assertTrue(accepted.getEntity() instanceof ImageGenerationView); + + assertRefusedAsNoSuchModel(CHAT_MODEL); + assertRefusedAsNoSuchModel(EMBEDDINGS_MODEL); + + // Exactly one call reached the provider: the accepted one. Neither refusal did. + wireMockServer.verify(1, postRequestedFor(urlPathEqualTo(IMAGES_PATH))); + } + + /** + * Given a request that omits {@code model} + * When the image is requested + * Then it is refused as a client error naming {@code model}, with no implicit default applied + */ + @Test + public void test_generations_withoutModel_isRejectedNamingTheField() { + final Response response = resource.generations( + mockRequest(host.getHostname(), bearerToken), mockResponse(), + host.getIdentifier(), generationFor(null, PROMPT)); + + assertNotNull(response); + assertEquals(400, response.getStatus()); + assertTrue(response.getEntity() instanceof InferenceErrorView); + + final InferenceErrorView.Body error = ((InferenceErrorView) response.getEntity()).error(); + assertNotNull(error); + assertEquals(ERROR_TYPE_INVALID_REQUEST, error.type()); + assertEquals("model", error.param()); + + wireMockServer.verify(0, postRequestedFor(urlPathEqualTo(IMAGES_PATH))); + } + + /** + * Given a request naming a configured model but omitting {@code prompt} + * When the image is requested + * Then it is refused as a client error naming {@code prompt}, and the provider is never + * contacted + * + *Named as its own field rather than folded into a generic "malformed request": there is + * nothing to generate without a prompt, and a caller sent back to guessing which of two + * required fields they missed is a support call.
+ */ + @Test + public void test_generations_withoutPrompt_isRejectedNamingTheField() { + final Response response = resource.generations( + mockRequest(host.getHostname(), bearerToken), mockResponse(), + host.getIdentifier(), generationFor(IMAGE_MODEL, null)); + + assertNotNull(response); + assertEquals(400, response.getStatus()); + assertTrue(response.getEntity() instanceof InferenceErrorView); + + final InferenceErrorView.Body error = ((InferenceErrorView) response.getEntity()).error(); + assertNotNull(error); + assertEquals(ERROR_TYPE_INVALID_REQUEST, error.type()); + assertEquals("prompt", error.param()); + + wireMockServer.verify(0, postRequestedFor(urlPathEqualTo(IMAGES_PATH))); + } + + /** + * Given a request carrying no credential at all + * When the image is requested + * Then it is refused as unauthorized and the provider is never contacted + * + *FR-015. Accepts a thrown {@link WebApplicationException} as well as a returned 401, since + * the surrounding authentication handshake may refuse before the resource body runs and either + * is a valid refusal.
+ */ + @Test + public void test_generations_withNoCredential_isUnauthorized() { + final HttpServletRequest request = mockRequest(host.getHostname(), null); + + try { + final Response response = resource.generations( + request, mockResponse(), host.getIdentifier(), + generationFor(IMAGE_MODEL, PROMPT)); + assertEquals("An anonymous request must be refused as unauthorized", + Response.Status.UNAUTHORIZED.getStatusCode(), response.getStatus()); + assertTrue("a refusal carries the standard error shape", + response.getEntity() instanceof InferenceErrorView); + } catch (final WebApplicationException e) { + assertEquals(Response.Status.UNAUTHORIZED.getStatusCode(), + e.getResponse().getStatus()); + } + + wireMockServer.verify(0, postRequestedFor(urlPathEqualTo(IMAGES_PATH))); + } + + /** + * Given a request asking for more images than the configured ceiling allows + * When the generation is requested + * Then it is refused with a 400 naming {@code n}, and nothing reaches the provider + * + *FR-012 and FR-037. Images are priced per image, so without a ceiling one accepted request + * multiplies a site's provider spend by whatever number it carried, and FR-032 puts per-site + * spend quotas out of scope — there is no second line of defence behind this one. The earlier + * draft that refused every {@code n} above 1 capped the spend by accident; correcting it to + * honor {@code n} removed that cap, and this is the deliberate replacement.
+ * + *Refused rather than clamped, and asserted as such: an implementation that quietly served + * {@link InferenceLimits#DEFAULT_MAX_IMAGES_PER_REQUEST} images here would return a 200 and a + * bill for ten, and the caller would never learn that the four hundred they asked for was not + * what they got. The zero provider calls are the other half — a clamp would show up here as a + * request that did reach the provider.
+ */ + @Test + public void test_generations_aboveTheImageCeiling_isRejectedNamingTheField() { + final int aboveCeiling = InferenceLimits.current().maxImagesPerRequest() + 1; + + final Response response = resource.generations( + mockRequest(host.getHostname(), bearerToken), mockResponse(), + host.getIdentifier(), generationFor(IMAGE_MODEL, PROMPT, aboveCeiling)); + + assertNotNull(response); + assertEquals("An n above the configured ceiling is one request carrying an unbounded bill, " + + "and must be refused rather than quietly clamped", 400, response.getStatus()); + assertTrue(response.getEntity() instanceof InferenceErrorView); + + final InferenceErrorView.Body error = ((InferenceErrorView) response.getEntity()).error(); + assertNotNull(error); + assertEquals(ERROR_TYPE_INVALID_REQUEST, error.type()); + assertEquals("The caller has to be told which field put them over the limit", + "n", error.param()); + assertNotNull(error.message()); + assertFalse(error.message().isBlank()); + + wireMockServer.verify(0, postRequestedFor(urlPathEqualTo(IMAGES_PATH))); + } + + /** + * Given a site whose image provider can only ever produce one image per request + * When several images are requested + * Then it is refused with a 400 naming {@code n} — not a retryable upstream failure + * + *FR-012. The multi-image call is a default method on the provider abstraction that throws + * unless the implementation overrides it: the OpenAI-backed models override it, the Gemini one + * does not. Left to the ordinary upstream translation of FR-031 that throw becomes a 502, which + * is the wrong answer in a way that costs the caller real time — 502 is retryable, so a + * standard client's back-off keeps re-sending a request that cannot succeed however long it + * waits. The status is the assertion that matters here; the param name is what lets the caller + * fix it on the first try.
+ * + *No provider stub is needed and none is reached: the site is configured against a provider + * that never answers in these tests, and the refusal is settled from the model itself before + * any call is made. That is also why this is safe to assert — it pins the behaviour to what the + * library actually implements rather than to a list of provider names in a test.
+ */ + @Test + public void test_generations_severalFromAModelThatCannotIsRefusedNotTreatedAsUpstreamFailure() + throws Exception { + AiTest.aiAppSecretsWithProviderConfig(host, geminiImageProviderConfigJson()); + + final Response response = resource.generations( + mockRequest(host.getHostname(), bearerToken), mockResponse(), + host.getIdentifier(), generationFor(GEMINI_IMAGE_MODEL, PROMPT, TWO_IMAGES)); + + assertNotNull(response); + assertEquals("A provider that can never honor n must be a 400 naming the field, not a " + + "retryable 502 that sends the caller's back-off into an unwinnable loop", + 400, response.getStatus()); + assertTrue(response.getEntity() instanceof InferenceErrorView); + + final InferenceErrorView.Body error = ((InferenceErrorView) response.getEntity()).error(); + assertNotNull(error); + assertEquals(ERROR_TYPE_INVALID_REQUEST, error.type()); + assertEquals("n", error.param()); + assertNotNull(error.message()); + assertFalse(error.message().isBlank()); + } + + /** + * @return a provider configuration whose image section names a provider that implements only + * the single-image call, leaving chat and embeddings pointed at the WireMock stub + */ + private static String geminiImageProviderConfigJson() { + return String.format( + "{" + + "\"chat\":{\"provider\":\"openai\",\"apiKey\":\"%1$s\",\"model\":\"%2$s\"," + + "\"endpoint\":\"%3$s\",\"maxRetries\":0}," + + "\"embeddings\":{\"provider\":\"openai\",\"apiKey\":\"%1$s\",\"model\":\"%4$s\"," + + "\"endpoint\":\"%3$s\",\"maxRetries\":0}," + + "\"image\":{\"provider\":\"google_ai\",\"apiKey\":\"%1$s\",\"model\":\"%5$s\"," + + "\"maxRetries\":0}" + + "}", + AiTest.API_KEY, CHAT_MODEL, String.format("http://localhost:%d/", AiTest.PORT), + EMBEDDINGS_MODEL, GEMINI_IMAGE_MODEL); + } + + /** + * Asserts that an {@code n} below 1 is refused here — as a 400 in the standard error shape, + * naming the field, with nothing reaching the provider. + * + *Shared by the values that differ only in how far below 1 they fall, so that both stay + * pinned to the same param name and the same status: a caller who asked for no images and one + * who asked for minus one have made the same mistake and deserve the same answer.
+ * + * @param count the number of images to ask for; below 1, or this asserts the wrong thing + */ + private void assertCountRefusedNamingTheField(final int count) { + final Response response = resource.generations( + mockRequest(host.getHostname(), bearerToken), mockResponse(), + host.getIdentifier(), generationFor(IMAGE_MODEL, PROMPT, count)); + + assertNotNull(response); + assertEquals("An n of " + count + " is below 1, which is not a number of images anyone " + + "can be served", 400, response.getStatus()); + assertTrue(response.getEntity() instanceof InferenceErrorView); + + final InferenceErrorView.Body error = ((InferenceErrorView) response.getEntity()).error(); + assertNotNull(error); + assertEquals(ERROR_TYPE_INVALID_REQUEST, error.type()); + assertEquals("A caller asking for a number of images that is not a number of images must " + + "be told which field is the problem", "n", error.param()); + assertNotNull(error.message()); + assertFalse(error.message().isBlank()); + + wireMockServer.verify(0, postRequestedFor(urlPathEqualTo(IMAGES_PATH))); + } + + /** + * Asserts that a model the site has configured for some other section is refused here. + * + * @param model the model to ask for + */ + private void assertRefusedAsNoSuchModel(final String model) { + final Response response = resource.generations( + mockRequest(host.getHostname(), bearerToken), mockResponse(), + host.getIdentifier(), generationFor(model, PROMPT)); + + assertNotNull(response); + assertEquals("'" + model + "' is not an image model, however well configured it is " + + "elsewhere on this site", 404, response.getStatus()); + assertTrue(response.getEntity() instanceof InferenceErrorView); + + final InferenceErrorView.Body error = ((InferenceErrorView) response.getEntity()).error(); + assertNotNull(error); + assertEquals(ERROR_TYPE_INVALID_REQUEST, error.type()); + assertEquals("model", error.param()); + assertNotNull(error.message()); + assertTrue(error.message().contains(model)); + + // What the refusal must not leak: which model the site does generate with, where its + // provider lives, what key reaches it, or which site is behind the host name. + assertFalse(error.message().contains(IMAGE_MODEL)); + assertFalse(error.message().contains(AiTest.API_KEY)); + assertFalse(error.message().contains(String.valueOf(AiTest.PORT))); + assertFalse(error.message().contains(host.getHostname())); + assertFalse(error.message().contains(host.getIdentifier())); + } + + /** + * Stubs the OpenAI-compatible provider: one canned image for an ordinary request, a different + * one for a request that really carried {@link #ALTERNATE_SIZE}, and a two-entry answer for a + * request that really asked for {@link #TWO_IMAGES} of them — so that what the caller asked for + * is distinguishable from what the provider would have chosen on its own. + * + *The three are separated by priority and by a body match on the value itself, which can + * only appear in the provider request when the field was passed through. The two matched stubs + * cannot collide: the size request asks for one image and the two-image request asks for the + * default size, and neither value is a substring of the other. All three are registered above + * the default priority on purpose: WireMock also loads the checked-in mappings under + * {@code src/test/resources/mappings}, several of which answer + * {@code POST /images/generations}, and without explicit priorities this file would be + * asserting against whichever stub happened to win.
+ */ + private static void stubProvider() { + wireMockServer.stubFor(post(urlPathEqualTo(IMAGES_PATH)) + .atPriority(1) + .withRequestBody(containing(ALTERNATE_SIZE)) + .willReturn(aResponse() + .withStatus(200) + .withHeader("Content-Type", "application/json") + .withBody(PROVIDER_ALTERNATE_SIZE_RESPONSE))); + + wireMockServer.stubFor(post(urlPathEqualTo(IMAGES_PATH)) + .atPriority(1) + .withRequestBody(matchingJsonPath(TWO_IMAGES_MARKER)) + .willReturn(aResponse() + .withStatus(200) + .withHeader("Content-Type", "application/json") + .withBody(PROVIDER_TWO_IMAGES_RESPONSE))); + + wireMockServer.stubFor(post(urlPathEqualTo(IMAGES_PATH)) + .atPriority(2) + .willReturn(aResponse() + .withStatus(200) + .withHeader("Content-Type", "application/json") + .withBody(PROVIDER_RESPONSE))); + } + + /** + * @param model the model to ask for, or null to omit the field + * @param prompt the prompt to generate from, or null to omit the field + * @return the smallest well-formed image-generation request, asking for one image + */ + private static ImageGenerationRequestView generationFor(final String model, + final String prompt) { + return generationFor(model, prompt, 1); + } + + /** + * @param model the model to ask for, or null to omit the field + * @param prompt the prompt to generate from, or null to omit the field + * @param count how many images to ask for, or null to omit {@code n} entirely — which is + * why it is boxed: "not sent" is a value a caller can express and FR-012 has to + * answer with one image, and a primitive would silently turn it into the 0 that + * FR-012 refuses + * @return an image-generation request asking for that many images + */ + private static ImageGenerationRequestView generationFor(final String model, + final String prompt, + final Integer count) { + return generationFor(model, prompt, count, SIZE); + } + + /** + * @param model the model to ask for, or null to omit the field + * @param prompt the prompt to generate from, or null to omit the field + * @param count how many images to ask for, or null to omit {@code n} entirely + * @param size the size to ask for + * @return an image-generation request asking for that many images at that size + */ + private static ImageGenerationRequestView generationFor(final String model, + final String prompt, + final Integer count, + final String size) { + return new ImageGenerationRequestView(model, prompt, count, size); + } + + /** + * Builds a request arriving at a given host name, with or without a bearer credential. + * + *Request attributes are backed by a real map rather than left as mock no-ops, because both + * the authentication handshake and FR-020's site attribution publish through them, and a mock + * that forgot what was set on it would not behave like a servlet container.
+ * + * @param serverName the host name the request arrives on + * @param credential the {@code Authorization} header value, or null for no credential + * @return the mocked request + */ + private static HttpServletRequest mockRequest(final String serverName, + final String credential) { + final HttpServletRequest request = mock(HttpServletRequest.class); + final MapThe interesting assertion in this file is the negative one: there is no administrator + * exemption. A role-conditional passthrough is exactly the branching that let two shipped dotAI + * endpoints drift to opposite model policies, and it would make SC-007 unstatable as a blanket + * invariant. So an administrator naming an unconfigured model is refused with the same status and + * the same body as a caller with no privileges at all, and that equality is asserted directly + * rather than inferred from two separate tests.
+ * + *The provider is a WireMock server standing in for an OpenAI-compatible endpoint, wired in + * through the same dotAI app secrets the rest of the AI integration tests use, so an accepted + * model travels the real client path rather than a stubbed one.
+ */ +public class InferenceModelValidationTest { + + /** The chat model the site is configured with. */ + private static final String CHAT_MODEL = "gpt-4o-mini"; + + /** The second entry of the fallback chain the site is given in one of these tests. */ + private static final String SECONDARY_MODEL = "gpt-4o-secondary"; + + /** A model name no site in these tests ever configures. */ + private static final String UNCONFIGURED_MODEL = "some-other-vendors-model"; + + /** Path an OpenAI-compatible provider serves completions on. */ + private static final String COMPLETIONS_PATH = "/chat/completions"; + + private static final String ERROR_TYPE_INVALID_REQUEST = "invalid_request_error"; + + /** What the stubbed provider answers whenever it is reached at all. */ + private static final String PROVIDER_RESPONSE = """ + { + "id": "chatcmpl-model-1", + "object": "chat.completion", + "created": 1789000000, + "model": "gpt-4o-mini", + "choices": [ + { + "index": 0, + "message": { + "role": "assistant", + "content": "Served." + }, + "finish_reason": "stop" + } + ], + "usage": {"prompt_tokens": 9, "completion_tokens": 2, "total_tokens": 11} + } + """; + + private static WireMockServer wireMockServer; + + /** A CMS administrator; the caller a role-conditional exemption would have privileged. */ + private static User adminUser; + private static String adminToken; + + /** A backend user with no administrative role, granted nothing but read on the test site. */ + private static User limitedUser; + private static String limitedToken; + + private Host host; + private final ChatCompletionsResource resource = new ChatCompletionsResource(); + + @BeforeClass + public static void beforeClass() throws Exception { + IntegrationTestInitService.getInstance().init(); + IPUtils.disabledIpPrivateSubnet(true); + wireMockServer = AiTest.prepareWireMock(); + stubProvider(); + + // Both roles are needed for the administrator, not one: WebResource.checkRolePermissions + // matches DOTCMS_BACK_END_USER by key without walking inheritance, so being an admin does + // not imply it, and admin is what grants read on the site passed as an explicit override. + adminUser = new UserDataGen() + .roles(APILocator.getRoleAPI().loadBackEndUserRole(), + APILocator.getRoleAPI().loadCMSAdminRole()) + .nextPersisted(); + adminToken = bearerTokenFor(adminUser); + + // The counterpart: admitted by FR-016 as an authenticated backend user, and nothing more. + // Read on the site is granted per test, since the site is created per test. + limitedUser = new UserDataGen() + .roles(APILocator.getRoleAPI().loadBackEndUserRole()) + .nextPersisted(); + limitedToken = bearerTokenFor(limitedUser); + } + + @AfterClass + public static void afterClass() { + wireMockServer.stop(); + IPUtils.disabledIpPrivateSubnet(false); + } + + @Before + public void before() throws Exception { + host = new SiteDataGen().nextPersisted(); + AiTest.aiAppSecretsWithProviderConfig(host, AiTest.providerConfigJson(AiTest.PORT, CHAT_MODEL)); + grantSiteRead(host, limitedUser); + wireMockServer.resetRequests(); + } + + @After + public void after() throws Exception { + AiTest.removeAiAppSecrets(host); + } + + /** + * Given a site configured with one chat model + * When a completion names that model + * Then it is accepted and the provider serves it + */ + @Test + public void test_completions_withConfiguredModel_isAccepted() { + final Response response = resource.completions( + mockRequest(adminToken), mockResponse(), host.getIdentifier(), + completionFor(CHAT_MODEL)); + + assertNotNull(response); + assertEquals(200, response.getStatus()); + assertTrue(response.getEntity() instanceof ChatCompletionView); + + final ChatCompletionView view = (ChatCompletionView) response.getEntity(); + assertEquals(CHAT_MODEL, view.model()); + wireMockServer.verify(1, postRequestedFor(urlPathEqualTo(COMPLETIONS_PATH))); + } + + /** + * Given a site configured with one chat model + * When a completion names a model the site has not configured + * Then it is refused with a 404 in the "no such model" shape, the provider is never contacted, + * and the message repeats nothing but what the caller already sent + */ + @Test + public void test_completions_withUnconfiguredModel_isRefusedAsNoSuchModel() { + final Response response = resource.completions( + mockRequest(adminToken), mockResponse(), host.getIdentifier(), + completionFor(UNCONFIGURED_MODEL)); + + assertNotNull(response); + assertEquals(404, response.getStatus()); + assertTrue(response.getEntity() instanceof InferenceErrorView); + + final InferenceErrorView errorView = (InferenceErrorView) response.getEntity(); + assertNotNull(errorView.error()); + assertEquals(ERROR_TYPE_INVALID_REQUEST, errorView.error().type()); + assertEquals("model", errorView.error().param()); + + final String message = errorView.error().message(); + assertNotNull(message); + assertTrue(message.contains(UNCONFIGURED_MODEL)); + + // What the refusal must not leak: which models the site does run, where its provider + // lives, what key reaches it, or which site is behind the host name. + assertFalse(message.contains(CHAT_MODEL)); + assertFalse(message.contains(AiTest.API_KEY)); + assertFalse(message.contains(String.valueOf(AiTest.PORT))); + assertFalse(message.contains(host.getHostname())); + assertFalse(message.contains(host.getIdentifier())); + + wireMockServer.verify(0, postRequestedFor(urlPathEqualTo(COMPLETIONS_PATH))); + } + + /** + * Given a CMS administrator and a backend user with no administrative role, both able to read + * the same site + * When each names a model the site has not configured + * Then both are refused, with the same status and the same error body + * + *This is the SC-007 invariant stated as one assertion rather than two: no caller, of any + * role, can cause a model outside the site's configuration to be invoked. Asserting the two + * refusals are equal is what would catch an exemption added later, which two + * independent single-role tests would not.
+ */ + @Test + public void test_completions_withUnconfiguredModel_refusesAdministratorAndNonAdministratorAlike() { + final Response adminResponse = resource.completions( + mockRequest(adminToken), mockResponse(), host.getIdentifier(), + completionFor(UNCONFIGURED_MODEL)); + final Response limitedResponse = resource.completions( + mockRequest(limitedToken), mockResponse(), host.getIdentifier(), + completionFor(UNCONFIGURED_MODEL)); + + assertNotNull(adminResponse); + assertNotNull(limitedResponse); + assertEquals(404, adminResponse.getStatus()); + assertEquals(adminResponse.getStatus(), limitedResponse.getStatus()); + + assertTrue(adminResponse.getEntity() instanceof InferenceErrorView); + assertTrue(limitedResponse.getEntity() instanceof InferenceErrorView); + + final InferenceErrorView.Body adminError = + ((InferenceErrorView) adminResponse.getEntity()).error(); + final InferenceErrorView.Body limitedError = + ((InferenceErrorView) limitedResponse.getEntity()).error(); + + assertEquals(adminError.type(), limitedError.type()); + assertEquals(adminError.param(), limitedError.param()); + assertEquals(adminError.message(), limitedError.message()); + assertEquals(adminError.code(), limitedError.code()); + + // Neither role reached the provider. + wireMockServer.verify(0, postRequestedFor(urlPathEqualTo(COMPLETIONS_PATH))); + } + + /** + * Given a request that omits {@code model} entirely + * When the completion is requested + * Then it is refused as a client error naming {@code model}, with no implicit default applied + */ + @Test + public void test_completions_withoutModel_isRejectedNamingTheField() { + final ChatCompletionRequestView requestView = new ChatCompletionRequestView( + null, + List.of(new MessageView("user", "Say something.", null, null, null)), + null, null, null, null, null, null, null, null, null, null); + + final Response response = resource.completions( + mockRequest(adminToken), mockResponse(), host.getIdentifier(), requestView); + + assertNotNull(response); + assertEquals(400, response.getStatus()); + assertTrue(response.getEntity() instanceof InferenceErrorView); + + final InferenceErrorView errorView = (InferenceErrorView) response.getEntity(); + assertNotNull(errorView.error()); + assertEquals(ERROR_TYPE_INVALID_REQUEST, errorView.error().type()); + assertEquals("model", errorView.error().param()); + + wireMockServer.verify(0, postRequestedFor(urlPathEqualTo(COMPLETIONS_PATH))); + } + + /** + * Given a site configured with a fallback chain of two models + * When a completion names the second entry of that chain + * Then it is accepted, not refused as unconfigured + * + *The set of accepted names is the whole chain, not its head. What actually serves is a + * separate question the site's configuration answers, not the caller: the client walks the + * chain from the front, so the served model reported back is an entry of the chain and need + * not be the one that was asked for.
+ */ + @Test + public void test_completions_withFallbackChainSecondEntry_isAccepted() throws Exception { + final ListFR-024 removed every reserved alias and every implicit default from this family, which makes + * this endpoint load-bearing rather than decorative: the list it returns is the only way + * a caller can learn which {@code model} values the chat endpoint will accept. That is what the + * tests here pin down — that the list is complete, that it is exactly the configured set with + * nothing invented, that it is drawn from the site the request resolved to, and that an + * unconfigured instance says so with an empty list rather than with somebody else's models.
+ * + *This endpoint contacts no provider — it reads configuration and nothing else. The WireMock + * server is started only because the dotAI app secrets these tests save must name an endpoint, + * and no stub on it is ever expected to be hit.
+ */ +public class InferenceModelsTest { + + /** The chat model the site created for every test is configured with. */ + private static final String CHAT_MODEL = "gpt-4o-mini"; + + /** A second, distinct chat model, used to tell one site's configuration from another's. */ + private static final String OTHER_CHAT_MODEL = "claude-sonnet-4-6"; + + /** The primary of a fallback chain — the entry a caller wanting "whatever the site runs" takes. */ + private static final String CHAIN_PRIMARY = "gpt-4o-mini"; + + /** The fallback of that chain; a name the chat endpoint accepts just as readily. */ + private static final String CHAIN_FALLBACK = "gpt-4o"; + + /** A third chain entry, so "the count matches" is not satisfied by a coincidence of two. */ + private static final String CHAIN_LAST_RESORT = "gpt-4o-mini-2024-07-18"; + + /** An embeddings model, configured to prove it does not leak into the chat listing. */ + private static final String EMBEDDINGS_MODEL = "text-embedding-3-small"; + + /** An image model, configured for the same reason. */ + private static final String IMAGE_MODEL = "dall-e-3"; + + /** Object type of the list envelope. */ + private static final String OBJECT_LIST = "list"; + + /** Object type of each entry in it. */ + private static final String OBJECT_MODEL = "model"; + + /** Owner every entry reports; dotCMS serves the model, whoever built it. */ + private static final String OWNED_BY = "dotcms"; + + private static WireMockServer wireMockServer; + private static User user; + private static String bearerToken; + + /** The site created for each test, configured with {@link #CHAT_MODEL}. */ + private Host host; + + /** Additional sites a test configured, torn down with it. */ + private final ListFR-010. Each entry of the chain is a name the completions endpoint accepts, so a list + * that showed only the primary would hide choices the caller is entitled to make.
+ */ + @Test + public void test_models_withFallbackChain_listsEveryEntryInOrder() throws Exception { + final Host chainSite = siteConfiguredWith(CHAIN_PRIMARY + "," + CHAIN_FALLBACK); + + final Response response = resource.models( + mockRequest(), mockResponse(), chainSite.getIdentifier()); + + assertNotNull(response); + assertEquals(200, response.getStatus()); + assertTrue(response.getEntity() instanceof ModelListView); + + final ModelListView view = (ModelListView) response.getEntity(); + assertEquals(OBJECT_LIST, view.object()); + assertEquals(List.of(CHAIN_PRIMARY, CHAIN_FALLBACK), modelIds(view)); + assertEquals("The primary model is the first entry", + CHAIN_PRIMARY, view.data().get(0).id()); + view.data().forEach( + (final ModelListView.ModelView model) -> assertEquals(OBJECT_MODEL, model.object())); + } + + /** + * Given a site configured with a three-entry chain and nothing else + * When the models are listed + * Then the list is exactly those three — no alias, no placeholder, no invented entry + * + *FR-024 removed the reserved alias precisely so the list equals the set of acceptable + * {@code model} values. An extra entry here would be a name the completions endpoint refuses, + * which is worse than no list at all.
+ */ + @Test + public void test_models_withConfiguredChain_addsNothingSynthetic() throws Exception { + final ListFR-022. The failure this guards against is not an error but a wrong success: falling + * back to whichever site happens to be configured would hand a caller model names their own + * site will refuse, and would disclose that some other site has dotAI set up.
+ */ + @Test + public void test_models_withNoConfigurationAnywhere_returnsEmptyList() { + // The system level is the only inheritance an unconfigured site has (ConfigService falls + // back to SYSTEM_HOST). If something else in this JVM has configured it, the condition + // under test does not hold and the assertion below would be meaningless rather than wrong. + Assume.assumeFalse("The system level must be unconfigured for this test to mean anything", + ConfigService.INSTANCE.config(APILocator.systemHost()).isEnabled()); + + final Host unconfiguredSite = new SiteDataGen().nextPersisted(); + + final Response response = resource.models( + mockRequest(), mockResponse(), unconfiguredSite.getIdentifier()); + + assertNotNull(response); + assertEquals(200, response.getStatus()); + assertTrue(response.getEntity() instanceof ModelListView); + + final ModelListView view = (ModelListView) response.getEntity(); + assertEquals(OBJECT_LIST, view.object()); + assertNotNull("An unconfigured site gets an empty list, not a null one", view.data()); + assertTrue("An unconfigured site must not be shown another site's models", + view.data().isEmpty()); + } + + /** + * Given two sites configured with different chat models + * When the models are listed with an explicit site override naming the second + * Then the second site's models come back and the first's do not + */ + @Test + public void test_models_withSiteOverride_listsTheResolvedSitesModels() throws Exception { + final Host otherSite = siteConfiguredWith(OTHER_CHAT_MODEL); + + final Response response = resource.models( + mockRequest(), mockResponse(), otherSite.getIdentifier()); + + assertNotNull(response); + assertEquals(200, response.getStatus()); + + final ModelListView view = (ModelListView) response.getEntity(); + assertEquals(List.of(OTHER_CHAT_MODEL), modelIds(view)); + assertFalse("The other site's model must not appear", + modelIds(view).contains(CHAT_MODEL)); + } + + /** + * Given an authenticated request carrying an explicit site override + * When the models are listed + * Then the resolved site is published on the request, so every response can report it + * + *FR-020. Asserted on the request attribute rather than on the header because invoking the + * resource method directly never runs the JAX-RS response filter that turns the attribute into + * {@code X-dotCMS-Resolved-Site} — the attribute is the part this endpoint is responsible + * for.
+ */ + @Test + public void test_models_withSiteOverride_publishesTheResolvedSiteId() { + final HttpServletRequest request = mockRequest(); + + final Response response = resource.models( + request, mockResponse(), host.getIdentifier()); + + assertNotNull(response); + assertEquals(200, response.getStatus()); + verify(request).setAttribute( + InferenceRequestAttributes.RESOLVED_SITE_ID, host.getIdentifier()); + } + + /** + * Given a request carrying no bearer token + * When the models are listed + * Then it is refused as unauthorized, in the standard error shape + * + *FR-015. The model list names the site's configured vendors and models, which is exactly + * the kind of reconnaissance an anonymous caller should not get for free.
+ */ + @Test + public void test_models_withoutBearerToken_isUnauthorized() { + final Response response = resource.models( + anonymousRequest(), mockResponse(), host.getIdentifier()); + + assertNotNull(response); + assertEquals(401, response.getStatus()); + assertTrue(response.getEntity() instanceof InferenceErrorView); + + final InferenceErrorView errorView = (InferenceErrorView) response.getEntity(); + assertNotNull(errorView.error()); + assertNotNull(errorView.error().message()); + assertFalse(errorView.error().message().isBlank()); + } + + /** + * Given a site configured with a chat model, a different embeddings model and a different + * image model + * When the models are listed + * Then only the chat model is listed + * + *The chat endpoint validates {@code model} against the chat section, so a list drawn from + * any other section would advertise names that endpoint refuses. Embeddings and images have + * their own operations and their own model sets.
+ */ + @Test + public void test_models_withDistinctSectionModels_listsOnlyTheChatSection() throws Exception { + final Host site = siteConfiguredWith(CHAT_MODEL, EMBEDDINGS_MODEL, IMAGE_MODEL); + + final Response response = resource.models( + mockRequest(), mockResponse(), site.getIdentifier()); + + assertNotNull(response); + assertEquals(200, response.getStatus()); + + final ModelListView view = (ModelListView) response.getEntity(); + assertEquals(List.of(CHAT_MODEL), modelIds(view)); + assertFalse("The embeddings model belongs to the embeddings operation, not here", + modelIds(view).contains(EMBEDDINGS_MODEL)); + assertFalse("The image model belongs to the image operation, not here", + modelIds(view).contains(IMAGE_MODEL)); + } + + /** + * Creates a site whose dotAI chat section names the given model, and registers it for teardown. + * + * @param chatModel the chat model, or a comma-separated fallback chain + * @return the configured site + */ + private Host siteConfiguredWith(final String chatModel) throws Exception { + return siteConfiguredWith(chatModel, AiTest.EMBEDDINGS_MODEL, AiTest.IMAGE_MODEL); + } + + /** + * Creates a site configuring each section independently, and registers it for teardown. + * + *Built here rather than with {@link AiTest#providerConfigJson(int, String)} because that + * helper fixes the embeddings and image models, and telling the sections apart is the point of + * one of these tests.
+ * + * @param chatModel the chat model, or a comma-separated fallback chain + * @param embeddingsModel the embeddings model + * @param imageModel the image model + * @return the configured site + */ + private Host siteConfiguredWith(final String chatModel, + final String embeddingsModel, + final String imageModel) throws Exception { + final Host site = new SiteDataGen().nextPersisted(); + AiTest.aiAppSecretsWithProviderConfig( + site, providerConfigJson(chatModel, embeddingsModel, imageModel)); + configuredSites.add(site); + return site; + } + + /** + * @param chatModel the chat model, or a comma-separated fallback chain + * @param embeddingsModel the embeddings model + * @param imageModel the image model + * @return the dotAI {@code providerConfig} JSON for a site configured that way + */ + private static String providerConfigJson(final String chatModel, + final String embeddingsModel, + final String imageModel) { + final String endpoint = String.format("http://localhost:%d/", AiTest.PORT); + return String.format( + "{" + + "\"chat\":{\"provider\":\"openai\",\"apiKey\":\"%s\",\"model\":\"%s\"," + + "\"endpoint\":\"%s\",\"maxRetries\":0}," + + "\"embeddings\":{\"provider\":\"openai\",\"apiKey\":\"%s\",\"model\":\"%s\"," + + "\"endpoint\":\"%s\",\"maxRetries\":0}," + + "\"image\":{\"provider\":\"openai\",\"apiKey\":\"%s\",\"model\":\"%s\"," + + "\"endpoint\":\"%s\",\"maxRetries\":0}," + + "\"settings\":{\"listenerIndexer\":{\"default\":\"blog\"}}" + + "}", + AiTest.API_KEY, chatModel, endpoint, + AiTest.API_KEY, embeddingsModel, endpoint, + AiTest.API_KEY, imageModel, endpoint); + } + + /** + * @param view the listing + * @return the model ids it carries, in the order it carries them + */ + private static ListIsolation here is not enforced by a check anywhere; it is a property of one line, the cache + * key {@code host + ":" + providerConfigHash} in + * {@link com.dotcms.ai.client.langchain4j.LangChain4jAIClient}. That makes it exactly the kind of + * behaviour that survives refactoring right up until it silently does not: nothing throws when two + * sites start sharing a provider instance, and nothing throws when a rotated credential keeps + * being used until the cache's one-hour TTL expires. The failure mode is a site's key spending + * another site's budget, or a revoked key continuing to work, and neither shows up anywhere but + * here.
+ * + *So each site is given a provider at a different URL path on the same + * WireMock server, answering with its own text. Where a request landed is then a fact, not an + * inference: the path WireMock recorded says which site's configuration built the client that + * made it.
+ * + *Two mechanisms should each make this hold on their own, which is why the test asserts the + * outcome rather than either of them: the cache key carries the configuration's hash, so a + * changed configuration is a different key, and {@code AIAppListener} additionally flushes the + * site's entries when its secrets are saved. A regression in one is invisible while the other + * still works — but a regression in both is a revoked credential that keeps working for an + * hour, so what is pinned here is the observable result.
+ */ + @Test + public void test_completions_afterRotatingOneSitesConfiguration_usesTheNewOneAndLeavesTheOtherAlone() + throws Exception { + // Warm the cache: without a first request there is no stale instance to invalidate and the + // test would pass on an implementation that never invalidates anything. + assertEquals(ALPHA_ANSWER, contentOf(complete(alphaSite, ALPHA_MODEL))); + wireMockServer.verify(1, postRequestedFor(urlPathEqualTo(ALPHA_PATH))); + + configure(alphaSite, "alpha-rotated", ALPHA_MODEL); + wireMockServer.resetRequests(); + + assertEquals(ALPHA_ROTATED_ANSWER, contentOf(complete(alphaSite, ALPHA_MODEL))); + wireMockServer.verify(1, postRequestedFor(urlPathEqualTo(ALPHA_ROTATED_PATH))); + wireMockServer.verify(0, postRequestedFor(urlPathEqualTo(ALPHA_PATH))); + + // The other site's provider was never rebuilt and never moved. + assertEquals(BETA_ANSWER, contentOf(complete(betaSite, BETA_MODEL))); + wireMockServer.verify(1, postRequestedFor(urlPathEqualTo(BETA_PATH))); + } + + /** + * Given two configured sites + * When a completion is requested for each + * Then each publishes its own site as the one that served, and never the other's + * + *The serving site is what FR-020 puts on every response and what a reconciliation + * pipeline would bill against, so reporting a neighbour's id would be a billing error rather + * than a cosmetic one.
+ */ + @Test + public void test_completions_forEitherSite_neverReportsTheOtherSitesResolvedSiteId() { + final HttpServletRequest alphaRequest = mockRequest(); + final HttpServletRequest betaRequest = mockRequest(); + + resource.completions(alphaRequest, mockResponse(), + alphaSite.getIdentifier(), completionFor(ALPHA_MODEL)); + resource.completions(betaRequest, mockResponse(), + betaSite.getIdentifier(), completionFor(BETA_MODEL)); + + verify(alphaRequest).setAttribute( + InferenceRequestAttributes.RESOLVED_SITE_ID, alphaSite.getIdentifier()); + verify(alphaRequest, never()).setAttribute( + InferenceRequestAttributes.RESOLVED_SITE_ID, betaSite.getIdentifier()); + + verify(betaRequest).setAttribute( + InferenceRequestAttributes.RESOLVED_SITE_ID, betaSite.getIdentifier()); + verify(betaRequest, never()).setAttribute( + InferenceRequestAttributes.RESOLVED_SITE_ID, alphaSite.getIdentifier()); + } + + /** + * Runs one completion for a site and asserts only that it was served, so the tests above can + * assert what matters about it. + * + * @param site the site to bill + * @param model the model that site has configured + * @return the completed answer + */ + private ChatCompletionView complete(final Host site, final String model) { + final Response response = resource.completions( + mockRequest(), mockResponse(), site.getIdentifier(), completionFor(model)); + + assertNotNull(response); + assertEquals(200, response.getStatus()); + assertTrue(response.getEntity() instanceof ChatCompletionView); + return (ChatCompletionView) response.getEntity(); + } + + /** + * @param view a completed answer + * @return the assistant's text, which identifies the provider that produced it + */ + private static String contentOf(final ChatCompletionView view) { + assertNotNull(view.choices()); + assertEquals(1, view.choices().size()); + assertNotNull(view.choices().get(0).message()); + return view.choices().get(0).message().content(); + } + + /** + * Points a site's dotAI configuration at one of the stubbed providers, and waits until that + * configuration is the one being resolved — re-saving alone would leave a rotation test racing + * the secrets cache. + * + * @param site the site to configure + * @param pathSegment the provider's URL path segment on the WireMock server + * @param model the chat model the site accepts + */ + private static void configure(final Host site, final String pathSegment, final String model) + throws Exception { + final String json = providerConfigJson(pathSegment, model); + AiTest.aiAppSecretsWithProviderConfig(site, json); + await().atMost(5, SECONDS).until(() -> { + final String resolved = ConfigService.INSTANCE.config(site).getProviderConfig(); + return resolved != null && resolved.contains("/" + pathSegment + "/"); + }); + } + + /** + * Builds a chat-only dotAI provider configuration whose endpoint is distinct per site, so the + * path a request arrives on names the configuration that produced it. + * + * @param pathSegment the provider's URL path segment on the WireMock server + * @param model the chat model + * @return the {@code providerConfig} JSON + */ + private static String providerConfigJson(final String pathSegment, final String model) { + return String.format( + "{\"chat\":{\"provider\":\"openai\",\"apiKey\":\"%s\",\"model\":\"%s\"," + + "\"endpoint\":\"http://localhost:%d/%s/\",\"maxRetries\":0}}", + AiTest.API_KEY, model, AiTest.PORT, pathSegment); + } + + /** + * Stubs three OpenAI-compatible providers on one server, each on its own path and each + * answering with text that names it. + */ + private static void stubProviders() { + stubProvider(ALPHA_PATH, ALPHA_ANSWER); + stubProvider(BETA_PATH, BETA_ANSWER); + stubProvider(ALPHA_ROTATED_PATH, ALPHA_ROTATED_ANSWER); + } + + private static void stubProvider(final String path, final String answer) { + wireMockServer.stubFor(post(urlPathEqualTo(path)) + .willReturn(aResponse() + .withStatus(200) + .withHeader("Content-Type", "application/json") + .withBody(providerResponse(answer)))); + } + + /** + * @param answer the assistant text this provider replies with + * @return an OpenAI-compatible completion payload carrying it + */ + private static String providerResponse(final String answer) { + return String.format(""" + { + "id": "chatcmpl-isolation", + "object": "chat.completion", + "created": 1789000000, + "model": "stubbed", + "choices": [ + { + "index": 0, + "message": { + "role": "assistant", + "content": "%s" + }, + "finish_reason": "stop" + } + ], + "usage": {"prompt_tokens": 9, "completion_tokens": 6, "total_tokens": 15} + } + """, answer); + } + + /** + * @param model the model to ask for + * @return the smallest valid completion request + */ + private static ChatCompletionRequestView completionFor(final String model) { + return new ChatCompletionRequestView( + model, + List.of(new MessageView("user", "Say something.", null, null, null)), + null, null, null, null, null, null, null, null, null, null); + } + + /** + * @return a request authenticated with the test user's bearer token, as the family requires + */ + private static HttpServletRequest mockRequest() { + final HttpServletRequest request = mock(HttpServletRequest.class); + when(request.getRequestURI()).thenReturn("/api/inference/v1/chat/completions"); + when(request.getRequestURL()) + .thenReturn(new StringBuffer("http://localhost/api/inference/v1/chat/completions")); + when(request.getMethod()).thenReturn("POST"); + when(request.getRemoteAddr()).thenReturn("127.0.0.1"); + when(request.getHeader("Authorization")).thenReturn(bearerToken); + return request; + } + + private static HttpServletResponse mockResponse() { + return mock(HttpServletResponse.class); + } +} diff --git a/dotcms-integration/src/test/java/com/dotcms/inference/rest/InferenceSiteResolutionTest.java b/dotcms-integration/src/test/java/com/dotcms/inference/rest/InferenceSiteResolutionTest.java new file mode 100644 index 000000000000..bf2fca40bb28 --- /dev/null +++ b/dotcms-integration/src/test/java/com/dotcms/inference/rest/InferenceSiteResolutionTest.java @@ -0,0 +1,474 @@ +package com.dotcms.inference.rest; + +import com.dotcms.ai.AiTest; +import com.dotcms.auth.providers.jwt.beans.ApiToken; +import com.dotcms.datagen.SiteDataGen; +import com.dotcms.datagen.UserDataGen; +import com.dotcms.inference.rest.view.ChatCompletionRequestView; +import com.dotcms.inference.rest.view.ChatCompletionRequestView.MessageView; +import com.dotcms.inference.rest.view.ChatCompletionView; +import com.dotcms.inference.rest.view.InferenceErrorView; +import com.dotcms.util.IntegrationTestInitService; +import com.dotcms.util.network.IPUtils; +import com.dotmarketing.beans.Host; +import com.dotmarketing.business.APILocator; +import com.github.tomakehurst.wiremock.WireMockServer; +import com.liferay.portal.model.User; +import org.junit.After; +import org.junit.AfterClass; +import org.junit.Before; +import org.junit.BeforeClass; +import org.junit.Test; + +import javax.servlet.http.HttpServletRequest; +import javax.servlet.http.HttpServletResponse; +import javax.ws.rs.core.Response; +import java.time.Duration; +import java.time.Instant; +import java.util.Date; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.UUID; + +import static com.github.tomakehurst.wiremock.client.WireMock.aResponse; +import static com.github.tomakehurst.wiremock.client.WireMock.post; +import static com.github.tomakehurst.wiremock.client.WireMock.urlPathEqualTo; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertTrue; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.anyString; +import static org.mockito.Mockito.doAnswer; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; + +/** + * Specifies which site's dotAI credentials serve a request on {@code /api/inference/v1}, and how a + * caller may say so — User Story 3 of the OpenAI-compatible inference feature. + * + *Per-site credential governance is the reason to route AI through dotCMS at all, so the + * question these tests answer is never "did the call succeed" but "whose provider configuration + * paid for it". That is asserted structurally rather than by inspecting secrets: the two sites are + * configured with different chat model names, and + * {@link ChatCompletionsResource} refuses any model the resolved site has not configured. A 200 + * for site A's model is therefore proof that site A's configuration was the one consulted, and a + * 404 for site B's model on the same host name is proof that site B's was not.
+ * + *What is asserted for FR-020. The serving site reaches a real caller as the + * {@code X-dotCMS-Resolved-Site} response header, which {@link ResolvedSiteHeaderFilter} writes + * from the request attribute {@link InferenceRequestAttributes#RESOLVED_SITE_ID}. These tests call + * the resource method directly, so no JAX-RS response filter runs and no header exists to read. + * They therefore assert the request attribute — the filter's sole input, and the + * only part of the chain the resource is responsible for.
+ */ +public class InferenceSiteResolutionTest { + + /** Path an OpenAI-compatible provider serves completions on. */ + private static final String COMPLETIONS_PATH = "/chat/completions"; + + private static final String REQUEST_URI = "/api/inference/v1/chat/completions"; + + /** Header form of the site override, as {@link ChatCompletionsResource} reads it. */ + private static final String SITE_HEADER = "X-dotCMS-Site"; + + /** + * Legacy request parameter that elsewhere in dotCMS selects the current site. FR-025 requires + * this family to ignore it. + */ + private static final String LEGACY_HOST_ID_PARAM = "host_id"; + + /** + * The other legacy site-selecting request parameter. dotCMS reads it under the velocity + * variable name of the Host content type, {@link Host#HOST_VELOCITY_VAR_NAME}; both that + * spelling and the lower-case one a caller would more naturally type are stubbed, so the test + * cannot pass merely because it aimed at the wrong string. + */ + private static final String LEGACY_HOST_PARAM = Host.HOST_VELOCITY_VAR_NAME; + + private static final String LEGACY_HOST_PARAM_LOWERCASE = "host"; + + /** The chat model configured on the first site, and on no other. */ + private static final String MODEL_SITE_A = "gpt-4o-mini"; + + /** The chat model configured on the second site, and on no other. */ + private static final String MODEL_SITE_B = "gpt-4o"; + + /** What the stubbed provider answers, whichever site's configuration reached it. */ + private static final String PROVIDER_RESPONSE = """ + { + "id": "chatcmpl-site-1", + "object": "chat.completion", + "created": 1789000000, + "model": "gpt-4o-mini", + "choices": [ + { + "index": 0, + "message": {"role": "assistant", "content": "Answered."}, + "finish_reason": "stop" + } + ], + "usage": {"prompt_tokens": 10, "completion_tokens": 2, "total_tokens": 12} + } + """; + + private static WireMockServer wireMockServer; + private static User user; + private static String bearerToken; + + private Host siteA; + private Host siteB; + + private final ChatCompletionsResource resource = new ChatCompletionsResource(); + + @BeforeClass + public static void beforeClass() throws Exception { + IntegrationTestInitService.getInstance().init(); + IPUtils.disabledIpPrivateSubnet(true); + wireMockServer = AiTest.prepareWireMock(); + stubProvider(); + + // Two roles, not one. WebResource.checkRolePermissions matches DOTCMS_BACK_END_USER by + // key and does not walk inheritance, so being an administrator does not imply it; the + // administrator role is separately what grants READ on the sites used as explicit + // overrides here. Refusals for callers who lack either live in + // InferenceAuthorizationTest. + user = new UserDataGen() + .roles(APILocator.getRoleAPI().loadBackEndUserRole(), + APILocator.getRoleAPI().loadCMSAdminRole()) + .nextPersisted(); + final ApiToken apiToken = APILocator.getApiTokenAPI().persistApiToken( + user.getUserId(), + Date.from(Instant.now().plus(Duration.ofDays(1))), + APILocator.systemUser().getUserId(), + "127.0.0.1"); + bearerToken = "Bearer " + APILocator.getApiTokenAPI().getJWT(apiToken, user); + } + + @AfterClass + public static void afterClass() { + wireMockServer.stop(); + IPUtils.disabledIpPrivateSubnet(false); + } + + @Before + public void before() throws Exception { + siteA = new SiteDataGen().name(uniqueSiteName("a")).nextPersisted(); + siteB = new SiteDataGen().name(uniqueSiteName("b")).nextPersisted(); + AiTest.aiAppSecretsWithProviderConfig( + siteA, AiTest.providerConfigJson(AiTest.PORT, MODEL_SITE_A)); + AiTest.aiAppSecretsWithProviderConfig( + siteB, AiTest.providerConfigJson(AiTest.PORT, MODEL_SITE_B)); + wireMockServer.resetRequests(); + } + + @After + public void after() throws Exception { + AiTest.removeAiAppSecrets(siteA); + AiTest.removeAiAppSecrets(siteB); + } + + /** + * Given a request whose host name is the host name of a configured site, and no override + * When a completion is requested + * Then that site's configuration serves it — its own model is accepted while the other site's + * model is refused as unconfigured — and the serving site is reported as that site + */ + @Test + public void test_completions_withNoOverride_isServedByTheHostNameSite() { + final HttpServletRequest request = mockRequest(siteA.getHostname()); + + final Response response = resource.completions( + request, mockResponse(), null, completionFor(MODEL_SITE_A)); + + assertNotNull(response); + assertEquals(200, response.getStatus()); + assertTrue(response.getEntity() instanceof ChatCompletionView); + assertResolvedSite(request, siteA); + + // The other site's model on the same host name proves the configuration consulted was + // site A's and not site B's; a call that merely succeeded would prove neither. + final HttpServletRequest otherModelRequest = mockRequest(siteA.getHostname()); + final Response otherModelResponse = resource.completions( + otherModelRequest, mockResponse(), null, completionFor(MODEL_SITE_B)); + + assertNotNull(otherModelResponse); + assertEquals(404, otherModelResponse.getStatus()); + assertTrue(otherModelResponse.getEntity() instanceof InferenceErrorView); + assertResolvedSite(otherModelRequest, siteA); + } + + /** + * Given a request whose host name is site A's, carrying {@code siteId} naming site B + * When a completion is requested for the model only site B has configured + * Then site B's configuration serves it and site B is reported as the serving site + */ + @Test + public void test_completions_withSiteIdQueryParameter_overridesTheHostName() { + final HttpServletRequest request = mockRequest(siteA.getHostname()); + + final Response response = resource.completions( + request, mockResponse(), siteB.getIdentifier(), completionFor(MODEL_SITE_B)); + + assertNotNull(response); + assertEquals(200, response.getStatus()); + assertTrue(response.getEntity() instanceof ChatCompletionView); + assertResolvedSite(request, siteB); + } + + /** + * Given a request whose host name is site A's, carrying the {@code X-dotCMS-Site} header + * naming site B by identifier + * When a completion is requested for the model only site B has configured + * Then site B's configuration serves it and site B is reported as the serving site + */ + @Test + public void test_completions_withSiteHeader_overridesTheHostName() { + final HttpServletRequest request = mockRequest(siteA.getHostname()); + when(request.getHeader(SITE_HEADER)).thenReturn(siteB.getIdentifier()); + + final Response response = resource.completions( + request, mockResponse(), null, completionFor(MODEL_SITE_B)); + + assertNotNull(response); + assertEquals(200, response.getStatus()); + assertTrue(response.getEntity() instanceof ChatCompletionView); + assertResolvedSite(request, siteB); + } + + /** + * Given a request carrying both overrides, the header naming site B and the query parameter + * naming site A + * When a completion is requested for the model only site B has configured + * Then the header wins: site B serves the request and is reported as the serving site + */ + @Test + public void test_completions_withHeaderAndQueryParameterDisagreeing_prefersTheHeader() { + final HttpServletRequest request = mockRequest(siteA.getHostname()); + when(request.getHeader(SITE_HEADER)).thenReturn(siteB.getIdentifier()); + + final Response response = resource.completions( + request, mockResponse(), siteA.getIdentifier(), completionFor(MODEL_SITE_B)); + + assertNotNull(response); + assertEquals(200, response.getStatus()); + assertTrue(response.getEntity() instanceof ChatCompletionView); + assertResolvedSite(request, siteB); + } + + /** + * Given a request whose host name is site A's, carrying the legacy {@code host_id} request + * parameter naming site B + * When a completion is requested for the model only site A has configured + * Then the legacy parameter is ignored: site A serves the request and is reported as the + * serving site + * + *FR-025. The parameter is a pre-existing de-facto site override on the rest of the + * product, and a token-authenticated API must not inherit it silently — a caller who never + * asked for another site must not be able to spend its credentials by copying a query string. + * The model is site A's, so honouring the parameter cannot merely look like success: it would + * surface as a 404 for a model site B never configured.
+ */ + @Test + public void test_completions_withLegacyHostIdParameter_ignoresIt() { + final HttpServletRequest request = mockRequest(siteA.getHostname()); + when(request.getParameter(LEGACY_HOST_ID_PARAM)).thenReturn(siteB.getIdentifier()); + + final Response response = resource.completions( + request, mockResponse(), null, completionFor(MODEL_SITE_A)); + + assertNotNull(response); + assertEquals("The legacy host_id parameter must not select the site; site A's model must " + + "still be accepted", 200, response.getStatus()); + assertTrue(response.getEntity() instanceof ChatCompletionView); + assertResolvedSite(request, siteA); + } + + /** + * Given a request whose host name is site A's, carrying the legacy {@code Host} request + * parameter naming site B + * When a completion is requested for the model only site A has configured + * Then the legacy parameter is ignored: site A serves the request and is reported as the + * serving site + */ + @Test + public void test_completions_withLegacyHostParameter_ignoresIt() { + final HttpServletRequest request = mockRequest(siteA.getHostname()); + when(request.getParameter(LEGACY_HOST_PARAM)).thenReturn(siteB.getHostname()); + when(request.getParameter(LEGACY_HOST_PARAM_LOWERCASE)).thenReturn(siteB.getHostname()); + + final Response response = resource.completions( + request, mockResponse(), null, completionFor(MODEL_SITE_A)); + + assertNotNull(response); + assertEquals("The legacy Host parameter must not select the site; site A's model must " + + "still be accepted", 200, response.getStatus()); + assertTrue(response.getEntity() instanceof ChatCompletionView); + assertResolvedSite(request, siteA); + } + + /** + * Given a request whose host name matches no site or alias, carrying the legacy + * {@code host_id} request parameter naming site B + * When a completion is requested + * Then the legacy parameter is still ignored: the default site serves the request, exactly as + * it would have without the parameter + * + *FR-025 has no exception for the fallback path, and it is the path that matters most. The + * unmatched host name is not an edge case here — FR-020 keeps the default-site fallback + * precisely because the server-side callers this family exists for routinely arrive on + * internal DNS, container service names or {@code localhost}, none of which are site aliases. + * A legacy override that is ignored only while the host name happens to match is not ignored; + * it is ignored where it could do no harm and honoured where it could.
+ */ + @Test + public void test_completions_withLegacyHostIdParameterAndUnmatchedHostName_ignoresIt() + throws Exception { + final Host defaultSite = APILocator.getHostAPI().findDefaultHost(APILocator.systemUser(), false); + final HttpServletRequest request = mockRequest(unmatchedHostName()); + when(request.getParameter(LEGACY_HOST_ID_PARAM)).thenReturn(siteB.getIdentifier()); + + resource.completions(request, mockResponse(), null, completionFor(MODEL_SITE_A)); + + assertEquals("The legacy host_id parameter must not select the site on the default-site " + + "fallback path either; it named site B (" + siteB.getIdentifier() + ")", + defaultSite.getIdentifier(), + request.getAttribute(InferenceRequestAttributes.RESOLVED_SITE_ID)); + } + + /** + * Given a request whose host name matches no site or alias, carrying the legacy {@code Host} + * request parameter naming site B + * When a completion is requested + * Then the legacy parameter is still ignored and the default site serves the request + */ + @Test + public void test_completions_withLegacyHostParameterAndUnmatchedHostName_ignoresIt() + throws Exception { + final Host defaultSite = APILocator.getHostAPI().findDefaultHost(APILocator.systemUser(), false); + final HttpServletRequest request = mockRequest(unmatchedHostName()); + when(request.getParameter(LEGACY_HOST_PARAM)).thenReturn(siteB.getHostname()); + when(request.getParameter(LEGACY_HOST_PARAM_LOWERCASE)).thenReturn(siteB.getHostname()); + + resource.completions(request, mockResponse(), null, completionFor(MODEL_SITE_A)); + + assertEquals("The legacy Host parameter must not select the site on the default-site " + + "fallback path either; it named site B (" + siteB.getIdentifier() + ")", + defaultSite.getIdentifier(), + request.getAttribute(InferenceRequestAttributes.RESOLVED_SITE_ID)); + } + + /** + * Given a request that will be refused because the model is not configured for its site + * When the completion is requested + * Then the serving site is still reported + * + *FR-020 asks for the serving site on every response, not only the ones that + * worked. A refusal is precisely the case an operator reconciling spend needs attributed, and + * a value only the happy path published would not deliver it.
+ */ + @Test + public void test_completions_whenRefused_stillReportsTheServingSite() { + final HttpServletRequest request = mockRequest(siteA.getHostname()); + + final Response response = resource.completions( + request, mockResponse(), null, completionFor("a-model-nobody-configured")); + + assertNotNull(response); + assertEquals(404, response.getStatus()); + assertTrue(response.getEntity() instanceof InferenceErrorView); + assertResolvedSite(request, siteA); + } + + /** + * Asserts the site the resource published as having served the request. + * + * @param request the request the resource was called with + * @param expected the site expected to have served it + */ + private static void assertResolvedSite(final HttpServletRequest request, final Host expected) { + assertEquals("The resolved-site request attribute, which ResolvedSiteHeaderFilter turns " + + "into the X-dotCMS-Resolved-Site header, must name the serving site", + expected.getIdentifier(), + request.getAttribute(InferenceRequestAttributes.RESOLVED_SITE_ID)); + } + + /** Answers every completion with the same finished response; the site under test is the variable. */ + private static void stubProvider() { + wireMockServer.stubFor(post(urlPathEqualTo(COMPLETIONS_PATH)) + .willReturn(aResponse() + .withStatus(200) + .withHeader("Content-Type", "application/json") + .withBody(PROVIDER_RESPONSE))); + } + + /** + * @param model the model to ask for + * @return the smallest well-formed completion request + */ + private static ChatCompletionRequestView completionFor(final String model) { + return new ChatCompletionRequestView( + model, + List.of(new MessageView("user", "Which site is serving me?", null, null, null)), + null, null, null, null, null, null, null, null, null, null); + } + + /** + * @param prefix distinguishes the two sites + * @return a host name no other test or run can collide with + */ + private static String uniqueSiteName(final String prefix) { + return "inference-" + prefix + "-" + UUID.randomUUID() + ".dotcms.com"; + } + + /** + * @return a host name no site or alias can match, standing in for the internal DNS and + * container service names the server-side callers of this family actually arrive on + */ + private static String unmatchedHostName() { + return "no-such-site-" + UUID.randomUUID() + ".invalid"; + } + + /** + * Builds a bearer-authenticated request arriving at a given host name. + * + *Request attributes are backed by a real map rather than left as mock no-ops, because the + * resolved site is published as an attribute and reading it back is how these tests observe + * which site served.
+ * + * @param serverName the host name the request arrives on + * @return the mocked request + */ + private static HttpServletRequest mockRequest(final String serverName) { + final HttpServletRequest request = mock(HttpServletRequest.class); + final Map