improvement(external-endpoints): v2 versions with clean signatures + updated docs based on openapi spec - #5273
Open
icecrasher321 wants to merge 178 commits into
Open
improvement(external-endpoints): v2 versions with clean signatures + updated docs based on openapi spec#5273icecrasher321 wants to merge 178 commits into
icecrasher321 wants to merge 178 commits into
Conversation
…ership workflow edits via sockets, ui improvements
…ng improvements, posthog, secrets mutations
…ration, signup method feature flags, SSO improvements
…nts, secrets performance, polling refactors, drag resources in mothership
…y invalidation, HITL docs
…endar triggers, docs updates, integrations/models pages improvements
…ions, jira forms endpoints
…mat, logs performance improvements fix(csp): add missing analytics domains, remove unsafe-eval, fix workspace CSP gap (#4179) fix(landing): return 404 for invalid dynamic route slugs (#4182) improvement(seo): optimize sitemaps, robots.txt, and core web vitals across sim and docs (#4170) fix(gemini): support structured output with tools on Gemini 3 models (#4184) feat(brightdata): add Bright Data integration with 8 tools (#4183) fix(mothership): fix superagent credentials (#4185) fix(logs): close sidebar when selected log disappears from filtered list; cleanup (#4186)
v0.6.46: mothership streaming fixes, brightdata integration
…m integration, atlassian triggers
Brings the five new v2 resource families (MCP servers, skills, custom tools, folders, credentials) from #6150 onto the branch, alongside the boundary-ratchet fix. Both sides edited the ratchet script in different places; the combined baselines are re-verified below.
#6150 branched before #6134, so skill-lifecycle.ts imports @/lib/workflows/orchestration/types — the module #6134 moved to @/lib/core/orchestration/types. Git merged a file deletion on one side with a new file referencing it on the other: no textual conflict, broken build. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
# Conflicts: # apps/sim/app/api/table/[tableId]/columns/route.ts # apps/sim/app/api/table/[tableId]/route.ts # apps/sim/app/api/table/[tableId]/rows/[rowId]/route.ts # apps/sim/app/api/v1/tables/[tableId]/columns/route.ts # apps/sim/app/api/v1/tables/[tableId]/rows/[rowId]/route.ts # apps/sim/app/api/v1/tables/[tableId]/rows/upsert/route.ts # apps/sim/lib/copilot/tools/server/table/user-table.ts # apps/sim/lib/folders/lifecycle.ts # apps/sim/lib/table/service.ts # scripts/check-api-validation-contracts.ts
…ementation (#6154) * refactor(knowledge): make lib/knowledge/orchestration the single implementation Knowledge base create was implemented four times — the internal route, v1, v2, and the copilot tool — and the orchestration around the shared write had drifted. Extract it the same way lib/table/orchestration was: services write, orchestration decides which writes run, guards them, audits them, and returns a transport-neutral failure. Behavior converged, not preserved: - One chunking default (DEFAULT_CHUNKING_CONFIG). The agent defaulted minSize to 1 against the API's 100, so identical input produced differently-chunked knowledge bases depending on who created it. The agent path now chunks at 100. - Every successful mutation is audited inside the orchestration function. The copilot tool called recordAudit zero times, so agent-created knowledge bases, document uploads, updates and deletes left no audit trail at all. - Failures classify by class, not by message text. The knowledge service errors are OrchestrationError subclasses and storage-quota rejections throw a shared StorageLimitExceededError, replacing four separate message greps for "already exists" / "does not have permission" / "storage limit". delete_connector reported the opposite of what happened. It reached the route through an internal HTTP self-call that sent no query string, so the route's keep-documents default always applied while the agent told the user the documents had been removed. The self-call is gone — all four connector operations run in-process — and the orchestration returns the real counts. Also: - OrchestrationErrorCode gains 'payload_too_large' (413 / PAYLOAD_TOO_LARGE). Without it, dropping the storage-limit message match would have regressed the documented 413 on knowledge base create and document upload to a 500. - messageForOrchestrationError renders a route's own wording for an unclassified fault, so a driver's message no longer reaches the client on a 500. - v1 and v2 knowledge base update now forward actorUserId, which the service requires for a workspace move; both omitted it. - The connector DELETE route reads deleteDocuments through parseRequest. Its contract declared z.boolean(), which would have rejected the string a query param actually is. - Drop the 409 from POST /api/v2/knowledge/{id}/documents in the OpenAPI spec. Nothing on the upload path throws a conflict; it was only ever reachable by the message match this change removes. Behavior change worth noting: a v1/v2 PUT carrying only the workspaceId scope field and no actual updates now returns 400 rather than 200 with the unchanged knowledge base. Deliberately deferred: document update remains internal-only. Extracting performUpdateKnowledgeDocument makes exposing it on v1/v2 a contract and a route away, but that is a new public surface rather than part of this consolidation. * fix(knowledge): make connector create atomic and stop flattening failures Review round 1 on #6154. - Resolve the billing payer before the connector is committed, not after. A malformed attribution header rejected post-commit left a live connector behind a 500, and a retry created a duplicate plus duplicate sync work. Manual sync resolves before writing its audit for the same reason. - Let the source-config validator carry its own failure class. Collapsing every rejection to `validation` flattened the connector PATCH route's 401 (stale stored credential) and 409 (missing workspace context) into a 400. - Add `unauthorized` to OrchestrationErrorCode. It is the class that 401 was already expressing on this route, and the v2 vocabulary already had UNAUTHORIZED; only the shared union was missing it. - Report a knowledge base that exists but failed to archive as failed, with the reason, rather than as not found. The copilot delete loop folded every non-not-found failure into `notFound`, telling the user it was never there. - Route copilot failures through the same message helper the HTTP surfaces use, so an unclassified fault's raw text (a driver's failed SQL) no longer reaches the agent verbatim while the UI and public APIs get the generic wording.
* feat(api): expand the public v2 files surface Adds folder support, rename/restore, move, bulk archive, share, and content replace to /api/v2/files, so managing files by API no longer stops at upload + download + archive-one. Routes are thin: auth -> parse -> perform* -> serialize. Share and content replace get their orchestration extracted first so the session routes and the public ones cannot diverge on the effective-authType resolution, the EE public-sharing gate, or the storage-quota classification. Presigned upload stays session-only: presign does an advisory quota check and the real debit happens in the separate register step, so a caller that never registers leaves unaccounted bytes with no reaper. The buffered multipart path debits inside uploadWorkspaceFile's own transaction. * fix(files): classify folder and content failures instead of 500ing them Bugbot round 1. The v2 routes map errorCode straight to a status, so every manager failure that arrived unclassified became a 500 for what is really a caller-fixable 400 or 404. - Folder manager throws OrchestrationError: missing target/folder -> not_found, reparent cycle / self-parent / restore-into-archived-workspace -> validation. - File manager does the same for the in-transaction 'File not found' paths that the earlier pass missed. - updateWorkspaceFileContent's outer catch re-wrapped everything in a bare Error, which stripped the class off StorageLimitExceededError and the new not_found alike. It now rethrows a classified failure untouched and attaches cause to the generic wrap, so asOrchestrationError can still walk the chain. - Every remaining perform* gained the asOrchestrationError branch. - renameWorkspaceFile returned the pre-update read, so the v2 PATCH reported a stale updatedAt; it now returns the timestamp it actually wrote. Docs: upload auto-suffixes a duplicate name rather than rejecting it, matching the in-app uploader. The description claimed 409 and was simply wrong. * fix(files): surface a failed upload read-back as the real error getWorkspaceFile swallows a query failure and returns null unless throwOnError is set, so a transient blip on the post-upload read reported as 'file could not be read back'. Distinguish the two: a real null after a just-committed write is an invariant break, a query failure is itself. * revert(api): drop the dedicated v2 file-folder routes File folders already live in the shared folder table as resourceType 'file' (#6045 cut them over, #6051 dropped workspace_file_folders), and the remaining file-specific folder machinery is being folded into the generic folder engine. Publishing /api/v2/files/folders/** would pin that transitional split into a public contract we'd then have to keep or break. Files stay folder-aware — folderId/folderPath on the projection, folderId on upload, and the move route — because a folder id is a folder.id and survives the unification untouched. Folder management belongs on /api/v2/folders once that surface serves resourceType 'file'; until then there is no v2 way to enumerate file folders, which is the deliberate gap. The orchestration classification fixes stay: the internal routes and the copilot file-folder tools still call those perform* functions. * fix(files): classify upload failures instead of matching their wording Bugbot round 2. uploadWorkspaceFile had the same outer-catch rewrap that updateWorkspaceFileContent did, so a blown storage quota reached the route as a bare Error and the v2 handler recovered the status by substring-matching the message. Any rewording silently demoted a 413 to a 500. - uploadWorkspaceFile rethrows a classified failure untouched and attaches cause to the generic wrap. - FileConflictError is now an OrchestrationError('conflict'), so a duplicate name classifies like every other conflict. Its 'FILE_EXISTS' discriminator had no readers and is gone; the instanceof checks elsewhere still hold. - The v2 upload handler uses v2CaughtOrchestrationError, dropping all three string matches. Also documents that bulk-archive is best-effort: unknown or already-archived ids are skipped rather than failing the call, and deletedItems is what actually happened. That asymmetry with the single-id DELETE was undocumented.
#6189) * feat(api): add search, filtering, and sorting to the v2 list endpoints One convention across every v2 list, documented on lib/api/contracts/v2/shared.ts: `search` (case-insensitive substring on the resource's natural name field), `sortBy` + `sortOrder` (per-resource enum, never a free string), and enumerated resource-specific filters. Reuses the sortBy/sortOrder pair v2 logs and v2 knowledge-documents already ship rather than inventing a third dialect alongside the Logs filters and the Tables predicate grammar. Every filter and sort is pushed into SQL. GET /api/v2/files previously read the whole scope and sorted/sliced it in JS; it now goes through a new queryWorkspaceFiles that filters, orders, and bounds the page in one query. Cursors are stamped with the sort they were minted under, so replaying one under a different sort is a 400 instead of silently duplicated or skipped rows. * fix(api): validate v2 cursor key values and compare timestamps at ms precision Two review findings, fixed at the root by making a keyset key own its cursor codec instead of hand-writing a decoder per sort. Cursor key values are caller-controlled, and matching the sort stamp and key count was not enough: an unparseable timestamp or a non-numeric size reached the query as an Invalid Date or NaN and surfaced as a 500. Each key now type- checks its own value and rejects a cursor it cannot hold, which both routes render as the documented 400. Timestamp keys now order and compare on date_trunc('milliseconds', col). Postgres keeps microseconds and defaultNow() populates them, but a cursor value round-trips through a millisecond-only JS Date — comparing the raw column against the truncated value re-admitted the page's own last row, duplicating it and stalling pagination outright at a page size of one. Reachable today via workspace_files.updated_at, which insertFileMetadata leaves to defaultNow().
…6184) * feat(api): complete the v2 workflows resource with versions and CRUD Adds version listing/detail plus create, update, and delete to the v2 workflows surface, which previously covered only execution and deployment. - GET /api/v2/workflows/[id]/versions — cursor-paginated, newest first - GET /api/v2/workflows/[id]/versions/[version] — version + pinned state - POST /api/v2/workflows, PATCH and DELETE /api/v2/workflows/[id] All six delegate to the existing orchestration and persistence helpers; no new domain logic. * fix(api): check folder containment before lock state; reject malformed version cursors assertFolderMutable walks a folder's ancestor chain without filtering on workspace, so inspecting it before containment let a caller tell a locked folder in someone else's workspace (423) from a nonexistent one (400). Create and update now assert containment first, matching the ordering import-workflow.ts already uses. A version cursor that decodes to JSON without a numeric version filtered every row out and returned an empty page with nextCursor null, which reads as a clean end-of-list. Malformed cursors are now a 400. * refactor(api): page workflow versions in the persistence helper listWorkflowVersions read every version row and the route filtered and sliced the result in memory, so the response was bounded but the query was not. It now takes optional limit/afterVersion, turning the cursor into a real keyset query; the route asks for limit + 1 and only trims the has-more probe. Both params are optional, so the internal, v1 admin, and copilot callers are unchanged. Also restores the untouched GET handler in [id]/route.ts to its original formatting — collapsing its signature had re-indented the whole body and buried the actual additions in whitespace churn.
* feat(api): expand the public v2 tables surface
Adds 16 operations so a v2 caller can do what the internal surface can:
rename/move/lock a table, restore it, manage saved views, run enrichment
columns, look up rows, and import/export with observable job control.
Extracts lib/table/orchestration/import.ts (performTableCsvImport,
performCreateTableFromCsv) and lib/table/export-stream.ts from the
first-party routes, then repoints those routes at them, so v1 and v2
cannot drift on what an import or export actually does.
events/stream, metadata and dispatches stay internal — they are editor
state, not public API.
* fix(api): make v2 table PATCH all-or-nothing and name the lock in every 423
Greptile P1: PATCH applied locks, rename and move as three sequential
transactions, so a folder rejected mid-request left the earlier writes
persisted while the response reported failure — and the schema-changed
signal was skipped, leaving open clients on stale state. Every rejectable
condition now runs before the first write, and the signal fires whenever
anything did land.
Cursor: v2TableLockError dropped the lock kind, so async import, column
run, enrichment and table mutations returned a bare LOCKED. A table has
four independent locks, so the caller could not tell which to clear.
* fix(api): report the lock kind on classified 423s too, not just thrown ones
The previous commit named the lock only where the rejection was thrown and
caught at the route boundary. Where it instead arrives as a classified
`errorCode: 'locked'` outcome — delete table, delete row, update column,
and the table mutations — the kind was dropped, so those 423s stayed
unactionable while their neighbours improved.
The orchestration results now carry `lock`, and a shared
`v2TableOrchestrationError` renders both arrival paths into the same
`{ code, message, details: { lock } }` body. `details` is omitted rather
than sent null when the kind is unknown, so a caller branching on it sees
absence instead of a phantom value.
* fix(api): make async table imports observable, not just startable
`POST /import-async` pointed callers at `GET /api/v2/tables/jobs` to track
progress, but that endpoint filters to `type = 'export'` — imports are
derived onto the table itself, one write job at a time, and exports get a
separate list precisely because they are excluded from that derivation.
The public Table shape omitted those derived fields, so an async import
could be started and cancelled but never observed to completion, failure,
or progress. That is the gap the import/export/job-control set was meant
to close.
Table now carries `job` — id, type, status, rowsProcessed, error, or null
when idle — and the import-async docs point at the table rather than the
export list.
* feat(api): make v2 table PATCH state which operations landed on failure
Greptile held the PR at 4/5 on the residual non-atomicity and named two
acceptable resolutions: make PATCH atomic, or have the contract adopt and
expose partial-success explicitly. Atomicity would mean threading one
transaction through renameTable, moveTableToFolder and updateTableLocks —
three shared service functions with four non-test callers including the
first-party route and two copilot tools — and deferring their per-operation
audits to commit time. That is a refactor of shared write paths well
outside this PR.
So the contract states it instead. Every rejectable condition is already
pre-validated, so a failure here is a genuine fault; when one follows a
successful operation the error now carries `details.applied` listing what
is live. Absent when nothing applied, so its presence always means "these
changes took effect despite the error". Documented on the operation.
`v2ErrorForOrchestration` gained the optional `details` this needs.
* fix(api): make table lock flags read-only on the public v2 surface
The new PATCH /api/v2/tables/[tableId] accepted a `locks` object, gated
on workspace admin plus the table-locks feature. That still lets an API
key clear the guard placed there to stop it: `write` is the floor for
the endpoint, and admin keys are ordinary API keys, so a lock is no
longer a boundary the key cannot cross.
Locks stay readable on the table resource and enforcement is unchanged
(a locked verb still returns 423). Changing one is now a first-party
admin action only.
The v2 body is declared here rather than reusing the first-party
updateTableBodySchema, which keeps its `locks` field so the UI can still
toggle them. It is .strict(), so a request carrying `locks` is rejected
with a 400 naming the field instead of silently succeeding without
applying it.
* fix(api): keep reporting applied operations when the PATCH re-read fails
The composite table PATCH promises that `error.details.applied` names the
operations that are live despite an error, but `applied` was scoped
inside the try. A rename or move that committed and was then followed by
a throw in the final re-read — or a re-read finding the table archived —
returned a bare 500/404 with no details, telling the caller nothing had
landed. It would then retry into a duplicate-name conflict or repeat the
move.
`applied` is now function-scoped so every post-write exit carries it: the
404 on a missing re-read, a thrown lock error, a classified orchestration
error, and the generic 500. `v2TableLockError` gains the same
`extraDetails` parameter `v2TableOrchestrationError` already had.
* feat(api): add workflow group writes to the v2 tables surface
v2 exposed GET /groups but none of the writes, so the public API could
run an enrichment or workflow column and read its binding, but never
create one. A caller could add a plain data column and trigger the
machine; wiring the two together still required the UI.
Adds POST/PATCH/DELETE on /api/v2/tables/[tableId]/groups. The group is
the unit that fills columns — one group feeds several — so creating one
creates its output columns in the same call, matching the first-party
shape rather than inverting it onto the column endpoint.
Four departures from the first-party body, all public-surface concerns:
- group.id is optional and server-generated. The UI mints an id to render
optimistically; a public caller has no such need and a client-chosen id
is a collision waiting to happen.
- outputColumns[].workflowGroupId is dropped from the body and stamped
from the resolved group, so it cannot disagree with it.
- autoRun defaults to false. First-party defaults true so a UI add fills
cells immediately; here it would make one POST fan out a metered run
across every existing row.
- A group naming neither a workflowId (type manual) nor an enrichmentId
(type enrichment) is a 400 rather than a half-specified group the route
has to guess about.
Also rejects an outputColumns entry no group output feeds — the two
arrays are joined by column name, and the first-party client builds both
from one picker so it cannot desync, but a public caller can.
Workspace containment on workflowId is asserted before it is persisted,
on create and on any update that re-points the group; without it a table
becomes a way to invoke workflows the key cannot otherwise reach.
* improvement(api): make v2 table import and export async-only
Drops the three synchronous entry points: POST /tables/[tableId]/import,
POST /tables/import-csv, and GET /tables/[tableId]/export.
Sync import tied a write to the lifetime of an HTTP request. The body
*was* the data, so it carried a 10 MB cap that Next silently truncates
past — a partial import reporting success. It also had no job, so a
timeout mid-write left rows in place with nothing to poll and nothing to
cancel. The async path reads the file from storage instead: upload via
POST /api/v2/files for a key, start with POST /import-async, watch
GET /tables/[tableId] -> job, stop with POST /job/cancel.
Sync export carried no such hazard, but one shape per operation beats
two: with both removed the surface has exactly one way to move a table
in or out, and the CLI wraps the extra calls.
This also removes the last multipart handling in v2 tables. Those were
the only routes bypassing parseRequest — form fields were parsed by hand
against separate form schemas, outside the contract system every other
v2 write goes through.
Create-a-table-from-CSV is now two calls: POST /tables, then
/import-async with createColumns. csvImportModeSchema is append|replace,
so there is no single-call create.
Route baseline 1064 -> 1061.
* docs(api): correct the import-async note about upload size limits
The docstring claimed there is no synchronous upload endpoint and so no
request-body size cliff. Both are wrong: POST /api/v2/files is a
synchronous multipart upload with a 100 MB cap, and it is the only v2
upload path (presigned is deliberately absent).
What async-only actually bought: the cap went 10 MB -> 100 MB, it fails
on an explicit size check and a bounded body read rather than a proxy cap
that silently truncates, authorization completes before any body is
buffered, and the table write is a job that can be watched and cancelled.
* feat(api): unify file and table transfers
* improvement(api): make multipart transfers stateless
* fix(api): make table import completion retries idempotent
`GET /api/v2/tables` returned every table in the workspace in one response — it used the cursor envelope but hardcoded `nextCursor: null`, and had no `limit`. That was defensible when tables were only created through the UI; `POST /api/v2/tables` is public now, so a script can create them in bulk and the list has no way to ask for less. Adds `queryTables` alongside `listTables` rather than changing it, so the internal callers that genuinely want the whole scope are untouched — the same split `queryWorkspaceFiles` / `listWorkspaceFiles` already uses. Filter, order and slice all run in the query, so a `search` never costs a full-workspace read. A cursor whose values don't bind raises a validation error instead of being coerced to "no filter", which would have silently served page 1 under a resumed cursor. The keyset closes on `id` so a page boundary inside a run of equal names or timestamps stays stable. The shared `LimitQuery` doc component said "Maximum rows to return"; it now serves the table list too, so the wording is resource-neutral. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…points # Conflicts: # apps/sim/app/api/credentials/route.ts # apps/sim/app/api/workflows/[id]/execute/route.ts # apps/sim/app/api/workflows/[id]/executions/[executionId]/route.ts # apps/sim/lib/copilot/tools/handlers/management/manage-skill.ts # apps/sim/lib/credentials/orchestration/index.ts # apps/sim/lib/uploads/contexts/workspace/workspace-file-manager.ts # package.json # packages/db/migrations/meta/0280_snapshot.json # packages/db/migrations/meta/_journal.json # scripts/check-api-validation-contracts.ts
…-upload feat(knowledge): multipart knowledge document uploads
* feat(uploads): unify signed upload sessions * fix(uploads): preserve attachment storage semantics * feat(files): add authored file creation * fix(uploads): omit hoisted S3 metadata headers
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Introduces a new, parallel
/api/v2external API surface that standardizes theresponse envelope, pagination, error handling, and auth across every resource —
fixing the inconsistencies that had accreted in
/api/v1without breaking existingv1 consumers. v1 stays in place and untouched on the wire; v2 is where the breaking
improvements land. Also includes a set of safe, in-place v1 correctness fixes
surfaced during the audit, and a full API-reference docs rework.
Type of Change
Testing
Tested manually
Checklist