diff --git a/docs/sandbox-provider.md b/docs/sandbox-provider.md index d3945db62..f64ce2fab 100644 --- a/docs/sandbox-provider.md +++ b/docs/sandbox-provider.md @@ -130,7 +130,7 @@ The configuration adapter must be non-nil, including its concrete value. Every ` Preview and persistence use `providers.Normalize` and `providers.Describe`. `providers.DiscoverSelection` resolves omitted native values before commit, and the complete specification is validated again at persistence. `providers.ResolveChange` owns configuration inheritance, and comparisons use normalized selectors, so preview, retry and commit share the same defaults. The store owns transactions, credential encryption, generation fencing, resource ownership and generic object storage: only the adapter interprets `provider_config` and `provider_metadata`, and `provider_credential` holds ciphertext bound to the installation and generation. Retained generations keep their original public configuration and metadata and compose the current credential through the adapter, so a credential replacement never rewrites a retained selector. Database constraints check object structure, not the registration list. -A direct adapter with a credential verifies all retained generations and allocation references before a key is replaced. The common `sandbox.CallFence` excludes native calls and waits for helper completion, including calls whose callers timed out; execution invokes the prepared verification and fencing callbacks without branching on a vendor. +A direct adapter with a credential verifies all retained generations and allocation references before a key is replaced. The common `sandbox.CallFence` excludes native calls and waits for helper completion, including calls whose callers timed out; execution owns verification and fencing, selected by the setup’s credential requirement and the adapter’s declared verification support. Its runtime manager loads and prepares deployment setups and publishes a monotonic generation cache. The common `sandbox` generation router resolves every allocation to a direct adapter or a node proxy bound to its node and generation. Vendor deployment validation and SDK setup stay at the construction boundary, and construction never creates an Environment. For node-local adapters `sandbox.Built` returns the provider, probe, installation identity, backend fingerprint and specification digest, and a `Quiescent` check when a helper can outlive its caller; generation collection waits for it. The factory also returns its close function. `execution.RuntimeProvider` binds the adapter to its kind, installation ID, backend fingerprint, generation, mode and node ownership; the database owns the selection, and the in-memory copy is never another authority. Docker and microsandbox run on nodes, and E2B is constructed directly. The node proxy exposes checkpoint operations only for a backend whose registered declaration supports them, and common lifecycle code admits suspension through the checkpoint declaration, never through a provider name. diff --git a/docs/zh/sandbox-provider.md b/docs/zh/sandbox-provider.md index 1aaf60a2b..c9a409626 100644 --- a/docs/zh/sandbox-provider.md +++ b/docs/zh/sandbox-provider.md @@ -1,7 +1,7 @@ --- title: "添加 Sandbox Provider" source: docs/sandbox-provider.md -source_hash: af35eef1f8c73b311a6166a5aa0352c61f33c71c834803f2e7bab6e51be3d965 +source_hash: 56570ea0d859bdfc26a2de3ea85ad8a002f03f7b5376010e9e4e195092b2a52f --- **Sandbox Provider** 为 Core 管理的 Environment 提供计算资源,以及在其中启动 [Sandbox I/O 服务](#oac-sandbox-io)的有界引导流程;该服务是 Provider 启动的唯一进程。本指南说明如何添加 Provider,并作为 Core 驱动 Provider 的参考。接口为 [`SandboxProvider`](https://github.com/MiniMax-AI/OpenAgentCore/blob/main/services/core/internal/sandbox/sandbox_provider.go)。 @@ -132,7 +132,7 @@ configuration adapter 必须非 nil,包括其具体值。每个 `Configuration 预览和持久化使用 `providers.Normalize` 与 `providers.Describe`。`providers.DiscoverSelection` 在提交前解析省略的原生值,持久化时再次验证完整 specification。`providers.ResolveChange` 负责配置继承,比较使用 normalized selector,使预览、重试和提交共享默认值。store 负责事务、凭据加密、generation fencing、资源所有权和通用对象存储:仅 adapter 解释 `provider_config` 与 `provider_metadata`,`provider_credential` 保存绑定到安装实例与 generation 的密文。保留 generation 保持原公开配置和 metadata,通过 adapter 组合当前凭据,因此替换凭据不重写保留 selector。数据库约束检查对象结构,不检查注册列表。 -具有凭据的 direct adapter 在替换 key 前验证全部保留 generation 与 allocation reference。公共 `sandbox.CallFence` 排除原生调用并等待 helper 完成,包括调用方已超时的调用;execution 调用已准备的 verification 和 fencing callback,不按厂商分支。 +具有凭据的 direct adapter 在替换 key 前验证全部保留 generation 与 allocation reference。公共 `sandbox.CallFence` 排除原生调用并等待 helper 完成,包括调用方已超时的调用;execution 负责验证与隔离,由 setup 的凭据要求和 adapter 声明的验证支持决定是否执行。其 runtime manager 加载与准备部署 setup,并发布 generation 单调递增的缓存。公共 `sandbox` generation router 将每个 allocation 解析为 direct adapter,或绑定其 node 与 generation 的 node proxy。 厂商部署验证和 SDK setup 留在构造边界,构造不创建 Environment。node-local adapter 的 `sandbox.Built` 返回 provider、probe、installation identity、backend fingerprint 和 specification digest;helper 可能比调用方存活更久时,还返回 `Quiescent` 检查,generation 回收会等待它。factory 还返回 close 函数。`execution.RuntimeProvider` 将 adapter 绑定到 kind、installation ID、backend fingerprint、generation、mode 和 node ownership;选择由数据库负责,内存副本不构成另一权限来源。Docker 与 microsandbox 在 node 上运行,E2B 直接构造。node proxy 仅对注册声明支持 checkpoint 的 backend 暴露 checkpoint 操作,公共 lifecycle 通过 checkpoint 声明准入 suspension,不通过 provider name。 diff --git a/services/core/IMPLEMENTATION.md b/services/core/IMPLEMENTATION.md index 7eec3a264..df1a97280 100644 --- a/services/core/IMPLEMENTATION.md +++ b/services/core/IMPLEMENTATION.md @@ -35,7 +35,7 @@ Domain owners, each with its PostgreSQL adapter under `internal/persistence/post - `environmenttemplates` (`templatepg`): Environment Templates, their validation and default network, their sealed setup, initial files, Skills and Plugins, and the resolved Template that Session creation composes into its Environment. - `modelconfiguration` (`modelconfigurationpg`): each Harness's deployment default model configuration and its last-use observations. - `skills` (`skillpg`): Skills and their immutable versions: archive checks, the default and latest pointers, version selection and deletion, and each version's sealed archive. Session creation freezes selected versions inside its own transaction: `skillpg.LockSkills` locks the selected Skills in ID order, `skillpg.ReadVersionForFreeze` opens and verifies a version, and `sessions` selects each version with the `skills` rules. -- `deployment` and its subpackage `deployment/placement` (`deploymentpg`, `placementpg`): the sandbox deployment and its nodes: provider configuration and the sealed credential, the specification and retained generations, setup, update and switch, node enrollment, identity and authentication, generation configuration, capacity, presence, status, host history, reset, and the counts of nodes and sandboxes bound to the public address; and hosted runtime allocations: reservation with sandbox Serve authority, compute ownership and settlement, observation diagnostics, activity, compute phases, wake receipts and cleanup, with the reads that schedule, discover and authorize them. It interprets Sandbox Provider declarations through the `providers.Registry` it is given, whose lookups return typed errors. `cmd/server` builds that registry and calls it only to build a direct Provider and to discover a Provider's configuration; it takes each setup's mode and declared operations from the `Setup` that `deployment` returns. `deployment/placement` owns hosted admission and placement: `cmd/server` builds one `placement.Rules` from the registry and the public URL, both fixed while Core runs, and gives it to `deployment.Service` and `sessions.Service`; its pure decisions admit a hosted Session, choose its node, and admit an allocation's reserved node and a restore on it, and its errors keep one status, code and message through `writeSandboxError`. `placementpg` loads the facts those decisions read and applies them on the caller's transaction-bound queries; it has no Store or transaction runner. The only other caller is Session creation: `sessions` decides admission and placement with those rules over the `placementpg` participants inside the creation transaction. Deployment changes and allocation writes run through `deployment.ExecutionOperations` on `deploymentpg.NewExecution(lease, …)`, which the Worker receives as `execution.Owner.Deployment`. Each allocation write locks the owning Session, decides in `deployment`, applies in `deploymentpg` and prunes the Session's change journal in the same leased transaction; cleanup settles the Session through the `sessions` procedures on a `sessionpg.SessionTx` bound to that transaction. The Worker reads the deployment, prepares a selection's setup and records live-compute activity through the pooled `deployment.Service` in `execution.Dispatcher.Deployment`, and lists the Sessions a reset still has to archive, schedules node lifecycles and reads allocations through the pooled `deployment.Reader` in `execution.Dispatcher.DeploymentReader`; a pooled read carries no lease, and the leased write that follows it rechecks the allocation's owner. Node management and reads use the pooled `deploymentpg.Store`, which `cmd/server` also reads the owner epoch from and runs the host-history sampler on. Provider calls run outside transactions, and the final transaction rechecks the expected generation. Session archive crosses the Session and the deployment, so it is a `deployment.ExecutionOperations` operation: `ArchiveSession`, which the administrator's archive route calls through `api.Execution.SessionArchive`, and `ArchiveResetSession`, which a reset calls, lock the Session through `deploymentpg`'s `WithSessionArchive`, check the deployment's generation and the running reset in `deployment`, then expire the Environment and cancel its work through the `sessions` procedures, release its agent-host assignment while retaining its home and request allocation cleanup, and record the audit entry in one leased transaction. `deployment.ObservationResolver` resolves a Session's Runtime observation target for `runtimeobs` from `sessions.SessionReader` and `deployment.Reader`. +- `deployment` and its subpackage `deployment/placement` (`deploymentpg`, `placementpg`): the sandbox deployment and its nodes: provider configuration and the sealed credential, the specification and retained generations, setup, update and switch, node enrollment, identity and authentication, generation configuration, capacity, presence, status, host history, reset, and the counts of nodes and sandboxes bound to the public address; and hosted runtime allocations: reservation with sandbox Serve authority, compute ownership and settlement, observation diagnostics, activity, compute phases, wake receipts and cleanup, with the reads that schedule, discover and authorize them. It interprets Sandbox Provider declarations through the `providers.Registry` it is given, whose lookups return typed errors. `cmd/server` builds that registry and gives it to `execution`, whose runtime manager constructs direct Providers, discovers configuration, and owns setup loading, preparation, credential verification and the generation cache; it takes each setup’s mode and declared operations from the `Setup` that `deployment` returns. `deployment/placement` owns hosted admission and placement: `cmd/server` builds one `placement.Rules` from the registry and the public URL, both fixed while Core runs, and gives it to `deployment.Service` and `sessions.Service`; its pure decisions admit a hosted Session, choose its node, and admit an allocation's reserved node and a restore on it, and its errors keep one status, code and message through `writeSandboxError`. `placementpg` loads the facts those decisions read and applies them on the caller's transaction-bound queries; it has no Store or transaction runner. The only other caller is Session creation: `sessions` decides admission and placement with those rules over the `placementpg` participants inside the creation transaction. Deployment changes and allocation writes run through `deployment.ExecutionOperations` on `deploymentpg.NewExecution(lease, …)`, which the Worker receives as `execution.Owner.Deployment`. Each allocation write locks the owning Session, decides in `deployment`, applies in `deploymentpg` and prunes the Session's change journal in the same leased transaction; cleanup settles the Session through the `sessions` procedures on a `sessionpg.SessionTx` bound to that transaction. The Worker reads the deployment, prepares a selection's setup and records live-compute activity through the pooled `deployment.Service` in `execution.Dispatcher.Deployment`, and lists the Sessions a reset still has to archive, schedules node lifecycles and reads allocations through the pooled `deployment.Reader` in `execution.Dispatcher.DeploymentReader`; a pooled read carries no lease, and the leased write that follows it rechecks the allocation's owner. Node management and reads use the pooled `deploymentpg.Store`, which `cmd/server` also reads the owner epoch from and runs the host-history sampler on. Provider calls run outside transactions, and the final transaction rechecks the expected generation. Session archive crosses the Session and the deployment, so it is a `deployment.ExecutionOperations` operation: `ArchiveSession`, which the administrator's archive route calls through `api.Execution.SessionArchive`, and `ArchiveResetSession`, which a reset calls, lock the Session through `deploymentpg`'s `WithSessionArchive`, check the deployment's generation and the running reset in `deployment`, then expire the Environment and cancel its work through the `sessions` procedures, release its agent-host assignment while retaining its home and request allocation cleanup, and record the audit entry in one leased transaction. `deployment.ObservationResolver` resolves a Session's Runtime observation target for `runtimeobs` from `sessions.SessionReader` and `deployment.Reader`. - `coremetrics` (`coremetricspg`): the Core metrics PostgreSQL holds: the root Turn queue counts, the root Turn history, read from one read-only snapshot, and the database size. `cmd/server`'s Core metrics source adds them to the process, pool, Worker and daemon registry measurements. - `runtimehistory` (`runtimehistorypg`): Runtime history samples: the periodic export, scoped reads and retention, which also prunes node-host samples. `runtimehistory.Service` scopes a read to the Session's hosted Environment through `sessions.EnvironmentReader`. - `sessions` (`sessionpg`): Session use cases and reads. The pooled `sessions.Service` runs the use cases on `sessionpg.Store`, which implements `sessions.Storage`, and plain reads use `sessions.Reader`, which `sessionpg.Store` also implements, directly. These cover Sessions (their creation, reads, the change journal and stream snapshot, diagnostics, the frozen execution configuration, measured usage and archive state, metadata updates, deletion and the public write audit), Turns (their reads and the execution work scan), input admission (a public input batch, Environment input reservation and the expiry of one reservation, with the reads of a Turn's admitted inputs, a reservation and the Environment input work scan), the model provider a Session froze, which `sessionpg.Store` opens with the credential key, root Items and Subagents (their reads), Session Artifacts (their reads, deletion and the staging of a Turn's export), Environments (their reads, the initialization list and the frozen setup and initial files, which `sessionpg.Store` opens with the credential key), agent-host identities (their reads, which include a Session's execution binding and the execution device list, registration, authentication and heartbeats), sandbox enrollment, the administrator's views across Projects (a Project's asset counts and Sessions for the summary, and the Sessions whose Runtime the administrator observes), executor credentials (authentication and a Project's credential state through `sessions.ExecutorCredentialReader`, and issuance, rotation and revocation; the Core-key Project operations record their audit entry in the same transaction) and native installation authorization and claims, whose tokens `sessionpg.Store` signs with the credential key. Session creation runs in one pooled `sessions.CreationTx`: `sessions` validates the request, computes its retry identity, with the provider key fingerprinted by `sessionpg.Store` under the credential key, and orders the upsert, hosted admission, the Skill freeze, the sealed model provider, execution configuration, initial files and setup, the Environment, placement and the initial input; `FindSessionCreation` finds an earlier creation by its recorded intent. `cmd/server` builds one `sessionpg.Store` with the credential key and the Service with its `placement.Rules`, and wires the Service and the Store into `execution.Dispatcher.Sessions` and `SessionsReader`, into the api fields, into Runtime enrollment and into the daemon gateway; the Worker creates Sessions through the Service after its execution checks, waking the scheduler only after the commit, and stages Artifacts through it; `cmd/environment-key` builds one without the key, and the Service without placement rules, for its credential commands. Turn transitions, execution completion, which alone publishes or discards a Turn's staged Artifacts, the start of Artifact capture, function calls and their application receipts, the Turn execution journal, Environment initialization, connection observations and their reconciliation, Session device binding, file-write reservation and settlement, and the promotion, failure and bulk expiry of Environment input reservations run through `sessions.ExecutionOperations` on `sessionpg.NewExecution(lease)`, which the Worker receives as `execution.Owner.Sessions`. diff --git a/services/core/cmd/server/main.go b/services/core/cmd/server/main.go index 12b6db7ef..c931e1d51 100644 --- a/services/core/cmd/server/main.go +++ b/services/core/cmd/server/main.go @@ -169,11 +169,10 @@ func run(config processconfig.Config) error { } // Node callbacks run only once the HTTP server serves, after the Worker starts. var worker *execution.Worker - managedNodes := configureManagedNodes(deploymentService, deploymentStore, sandboxProviders, config, func(ctx context.Context) error { + managedNodes := configureManagedNodes(deploymentService, deploymentStore, func(ctx context.Context) error { return worker.CheckOwnership(ctx) }) - defer managedNodes.hub.Close() - observationSource := managedNodes.setup.observationSource + defer managedNodes.Close() observationResolver, err := deployment.NewObservationResolver(sessionStore, deploymentStore) if err != nil { return err @@ -182,22 +181,12 @@ func run(config processconfig.Config) error { if err != nil { return err } - observationService, err := runtimeobs.NewService(observationResolver, observationSource, history.Options...) - if err != nil { + defer func() { if history.Exporter != nil { closeCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second) defer cancel() closeRuntimeHistory(closeCtx, history.Exporter) } - return err - } - defer func() { - closeCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second) - defer cancel() - _ = observationService.Close(closeCtx) - if history.Exporter != nil { - closeRuntimeHistory(closeCtx, history.Exporter) - } }() if err := api.ValidateCredentialSeparation(ctx, config.CoreKeys, projectStore); err != nil { return err @@ -233,12 +222,13 @@ func run(config processconfig.Config) error { if buildRevision != "" { nativeInstaller = &api.NativeInstaller{Version: buildRevision, Base: config.PublicOrigin.InstallerBase(), Catalog: catalog} } + sandboxLink, _ := config.PublicOrigin.SandboxLink() dispatcher := &execution.Dispatcher{Registry: registry, Credentials: vaultService, Observer: modelConfigurationStore, Deployment: deploymentService, DeploymentReader: deploymentStore, - Sessions: sessionService, - SessionsReader: sessionStore, - Links: linkRelay, - ManagedRuntimes: managedNodes.runtime, MaxConcurrentExecutions: config.ExecutionConcurrency} + Sessions: sessionService, + SessionsReader: sessionStore, + Links: linkRelay, + Providers: sandboxProviders, NodeProviders: managedNodes, ProviderPaths: config.ProviderPaths, InstallationID: config.InstallationID, SandboxLink: sandboxLink, MaxConcurrentExecutions: config.ExecutionConcurrency} lease, err := pgunit.AcquireLease(ctx, pool) if err != nil { return err @@ -268,6 +258,15 @@ func run(config processconfig.Config) error { <-workerDone } }() + observationService, err := runtimeobs.NewService(observationResolver, worker.ObservationSource, history.Options...) + if err != nil { + return err + } + defer func() { + closeCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + _ = observationService.Close(closeCtx) + }() // The Worker owns every sampling sweep. sampler, err := runtimeobs.NewSampler(observationResolver, observationService, worker, runtimeobs.SamplerOptions{}) if err != nil { @@ -345,7 +344,7 @@ func run(config processconfig.Config) error { NodeAllocations: deploymentStore, DeploymentChanges: worker, DeploymentReset: worker, - ConfigurationDiscovery: managedNodes.setup, + ConfigurationDiscovery: worker, }, } apiHandler, err := api.NewHandler(deps) @@ -355,7 +354,7 @@ func run(config processconfig.Config) error { handler := serverHandler(apiHandler, &daemonRoutes{gateway: daemonHandler, enrollment: runtimeenrollment.EnrollmentHandler(sessionService, config.PublicOrigin), connection: connections, - nodeConnect: managedNodes.hub}) + nodeConnect: managedNodes}) server := &http.Server{Addr: config.Addr, Handler: handler, ReadHeaderTimeout: 10 * time.Second, ReadTimeout: 30 * time.Second, WriteTimeout: 30 * time.Second, IdleTimeout: 60 * time.Second} done := make(chan error, 1) go func() { done <- server.ListenAndServe() }() diff --git a/services/core/cmd/server/managed_generation_operations.go b/services/core/cmd/server/managed_generation_operations.go deleted file mode 100644 index 04019cc22..000000000 --- a/services/core/cmd/server/managed_generation_operations.go +++ /dev/null @@ -1,96 +0,0 @@ -package main - -import ( - "context" - "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/providercontract" - "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox" -) - -func (p *generationRouter) Initial(ctx context.Context, r sandbox.Reference) (sandbox.Compute, error) { - if err := providercontract.Require(p, "Initial"); err != nil { - return sandbox.Compute{}, err - } - v, done, err := p.route(ctx, r) - if err != nil { - return sandbox.Compute{}, err - } - defer done() - return v.Initial(ctx, r) -} -func (p *generationRouter) NewCompute(ctx context.Context, r sandbox.Reference, g uint64, snapshot *sandbox.SnapshotIdentity) (sandbox.Compute, error) { - if err := providercontract.Require(p, "NewCompute"); err != nil { - return sandbox.Compute{}, err - } - v, done, err := p.route(ctx, r) - if err != nil { - return sandbox.Compute{}, err - } - defer done() - return v.NewCompute(ctx, r, g, snapshot) -} -func (p *generationRouter) GetCompute(ctx context.Context, r sandbox.Reference, c sandbox.Compute) (sandbox.ComputeState, error) { - if err := providercontract.Require(p, "GetCompute"); err != nil { - return sandbox.ComputeState{}, err - } - v, done, err := p.route(ctx, r) - if err != nil { - return sandbox.ComputeState{}, err - } - defer done() - return v.GetCompute(ctx, r, c) -} -func (p *generationRouter) Suspend(ctx context.Context, q sandbox.SuspendRequest) (sandbox.ComputeState, error) { - if err := providercontract.Require(p, "Suspend"); err != nil { - return sandbox.ComputeState{}, err - } - v, done, err := p.route(ctx, q.Reference) - if err != nil { - return sandbox.ComputeState{}, err - } - defer done() - return v.Suspend(ctx, q) -} -func (p *generationRouter) Resume(ctx context.Context, q sandbox.ResumeRequest) (sandbox.ComputeState, error) { - if err := providercontract.Require(p, "Resume"); err != nil { - return sandbox.ComputeState{}, err - } - v, done, err := p.route(ctx, q.Reference) - if err != nil { - return sandbox.ComputeState{}, err - } - defer done() - return v.Resume(ctx, q) -} -func (p *generationRouter) KillCompute(ctx context.Context, r sandbox.Reference, c sandbox.Compute) error { - if err := providercontract.Require(p, "KillCompute"); err != nil { - return err - } - v, done, err := p.route(ctx, r) - if err != nil { - return err - } - defer done() - return v.KillCompute(ctx, r, c) -} -func (p *generationRouter) DeleteSnapshot(ctx context.Context, r sandbox.Reference, snapshot sandbox.SnapshotIdentity) error { - if err := providercontract.Require(p, "DeleteSnapshot"); err != nil { - return err - } - v, done, err := p.route(ctx, r) - if err != nil { - return err - } - defer done() - return v.DeleteSnapshot(ctx, r, snapshot) -} -func (p *generationRouter) ResumeCompute(ctx context.Context, r sandbox.Reference, c sandbox.Compute) (sandbox.ComputeState, error) { - if err := providercontract.Require(p, "ResumeCompute"); err != nil { - return sandbox.ComputeState{}, err - } - v, done, err := p.route(ctx, r) - if err != nil { - return sandbox.ComputeState{}, err - } - defer done() - return v.ResumeCompute(ctx, r, c) -} diff --git a/services/core/cmd/server/managed_generations.go b/services/core/cmd/server/managed_generations.go deleted file mode 100644 index 21c8ecd0d..000000000 --- a/services/core/cmd/server/managed_generations.go +++ /dev/null @@ -1,190 +0,0 @@ -package main - -import ( - "context" - "errors" - "maps" - "time" - - "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/deployment" - "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/providercontract" - - "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/execution" - "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/runtimeobs" - "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox" -) - -// Every facade, including already running lifecycles, resolves the allocation's -// immutable specification and current credential. There is no mutable provider -// map to unload and no current-generation fallback for a missing historical row. -type generationRouter struct { - setup *managedSetup - operations providercontract.Operations -} - -func (p *generationRouter) route(ctx context.Context, r sandbox.Reference) (sandbox.SandboxProvider, func(), error) { - release, err := p.setup.providerCalls.Enter(ctx) - if err != nil { - return nil, nil, err - } - setup, err := p.setup.deployment.AllocationSetup(ctx, r) - if err != nil { - release() - return nil, nil, err - } - provider, err := p.setup.provider(setup) - if err != nil { - release() - return nil, nil, err - } - return provider, release, nil -} -func (p *generationRouter) Create(ctx context.Context, b sandbox.Bootstrap) (sandbox.Info, error) { - v, done, err := p.route(ctx, b.Reference) - if err != nil { - return sandbox.Info{}, err - } - defer done() - return v.Create(ctx, b) -} -func (p *generationRouter) GetInfo(ctx context.Context, r sandbox.Reference) (sandbox.Info, error) { - v, done, err := p.route(ctx, r) - if err != nil { - return sandbox.Info{}, err - } - defer done() - return v.GetInfo(ctx, r) -} -func (p *generationRouter) Renew(ctx context.Context, r sandbox.Reference) (sandbox.Info, error) { - v, done, err := p.route(ctx, r) - if err != nil { - return sandbox.Info{}, err - } - defer done() - return v.Renew(ctx, r) -} -func (p *generationRouter) Kill(ctx context.Context, r sandbox.Reference) error { - v, done, err := p.route(ctx, r) - if err != nil { - return err - } - defer done() - return v.Kill(ctx, r) -} - -func (p *generationRouter) ProviderOperations() providercontract.Operations { - return maps.Clone(p.operations) -} -func (p *generationRouter) Observe(ctx context.Context, t runtimeobs.Target) (runtimeobs.Sample, error) { - v, done, err := p.route(ctx, sandbox.Reference{TenantID: t.TenantID, EnvironmentID: t.EnvironmentID, AllocationID: t.Instance.AllocationID}) - if err != nil { - return runtimeobs.Sample{}, err - } - defer done() - if err := providercontract.Require(v, "Observe"); err != nil { - return runtimeobs.Sample{}, err - } - return v.Observe(ctx, t) -} - -// routeGenerations routes a direct provider's allocations through their own -// generations. Node providers route through the node transport instead. -func (s *managedSetup) routeGenerations(candidate execution.PreparedRuntimeDeployment, setup deployment.Setup) (execution.PreparedRuntimeDeployment, error) { - if setup.Mode == "nodes" { - return candidate, nil - } - candidate.Config.Provider = &generationRouter{setup: s, operations: candidate.Config.Provider.ProviderOperations()} - if err := sandbox.ValidateProvider(candidate.Config.Provider); err != nil { - return execution.PreparedRuntimeDeployment{}, err - } - if !setup.UsesCredential { - return candidate, nil - } - candidate.FenceCredential = func(ctx context.Context) (func(), error) { - release, err := s.providerCalls.Fence(ctx) - if err != nil { - return nil, sandbox.ErrConfigurationUnconfirmed - } - return release, nil - } - candidate.VerifyCredential = func(ctx context.Context) error { - ctx, cancel := context.WithTimeout(ctx, 30*time.Second) - defer cancel() - // Preserve the committed credential until its ownership anchor is verified. - // Public template readability cannot establish which team owns a deployment. - verify := func(value deployment.Setup, refs []sandbox.Reference) error { - value.InstallationID = setup.InstallationID - return s.registry.VerifyCredential(ctx, s.direct(value), refs) - } - current, err := s.deployment.Setup(ctx) - if err != nil { - return err - } - if current.Provider != setup.Provider { - return &deployment.ResetRequiredError{CurrentProvider: current.Provider, RequestedProvider: setup.Provider} - } - if err := verify(current, nil); err != nil { - if errors.Is(err, sandbox.ErrCredentialRejected) || errors.Is(err, sandbox.ErrCredentialOwnership) { - // A revoked legacy key or a public template outside its team cannot - // anchor ownership. This says nothing about the candidate key's validity. - return &deployment.ResetRequiredError{CurrentProvider: setup.Provider, RequestedProvider: setup.Provider} - } - return err - } - withCandidateKey := func(value deployment.Setup, refs []sandbox.Reference) error { - value, err := s.deployment.WithCredential(value, setup) - if err != nil { - return err - } - return verify(value, refs) - } - if err := withCandidateKey(current, nil); err != nil { - return err - } - if err := verify(setup, nil); err != nil { - return err - } - generations := map[uint64]deployment.Setup{current.Generation: current} - for after := int64(-1); ; { - page, err := s.deployment.GenerationPage(ctx, after) - if err != nil { - return err - } - for _, g := range page { - generations[g.Generation] = g - if err := withCandidateKey(g, nil); err != nil { - return err - } - after = int64(g.Generation) - } - if len(page) < 32 { - break - } - } - for after := ""; ; { - page, err := s.allocations.CredentialAllocations(ctx, after) - if err != nil { - return err - } - refsByGeneration := make(map[uint64][]sandbox.Reference) - for _, a := range page { - refsByGeneration[a.DeploymentGeneration] = append(refsByGeneration[a.DeploymentGeneration], sandbox.Reference{TenantID: a.TenantID, EnvironmentID: a.EnvironmentID, AllocationID: a.ID}) - after = a.ID - } - for generation, refs := range refsByGeneration { - owner, ok := generations[generation] - if !ok { - return sandbox.ErrConfigurationUnconfirmed - } - if err := withCandidateKey(owner, refs); err != nil { - return err - } - } - if len(page) < 32 { - break - } - } - return nil - } - return candidate, nil -} diff --git a/services/core/cmd/server/managed_nodes.go b/services/core/cmd/server/managed_nodes.go index 8383abb35..74daae230 100644 --- a/services/core/cmd/server/managed_nodes.go +++ b/services/core/cmd/server/managed_nodes.go @@ -5,27 +5,17 @@ import ( "errors" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/deployment" - "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/execution" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/persistence/postgres/pgunit" - "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/processconfig" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox/node" - "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox/providers" ) -type managedNodes struct { - setup *managedSetup - runtime *execution.RuntimeProvider - hub *node.Hub -} - // configureManagedNodes serves the nodes of the Web-managed deployment. Node // presence and health and the generation of each allocation go through the // deployment service; the owner epoch that fences connections and the // allocations each generation retains are read from the deployment reader. -func configureManagedNodes(nodes *deployment.Service, reader deployment.Reader, registry *providers.Registry, config processconfig.Config, owner func(context.Context) error) *managedNodes { - result := &managedNodes{} - result.hub = node.NewHub(node.HubOptions{ +func configureManagedNodes(nodes *deployment.Service, reader deployment.Reader, owner func(context.Context) error) *node.Hub { + return node.NewHub(node.HubOptions{ Generations: func(ctx context.Context, n node.Identity, connection string, epoch uint64, health node.Health) error { if err := owner(ctx); err != nil { return err @@ -70,12 +60,6 @@ func configureManagedNodes(nodes *deployment.Service, reader deployment.Reader, return nodes.Heartbeat(ctx, n.NodeID, connection, epoch, nodeHealthRecord(health)) }, }) - // An origin without a sandbox Link admits no hosted sandbox. - link, _ := config.PublicOrigin.SandboxLink() - result.setup = &managedSetup{processPaths: config.ProviderPaths, registry: registry, deployment: nodes, allocations: reader, hub: result.hub, installationID: config.InstallationID, sandboxLink: link} - result.runtime = execution.NewDeferredRuntimeProvider(config.InstallationID, result.setup.load, result.setup.prepare) - result.runtime.PublishUnconfigured = result.setup.publishUnconfigured - return result } func nodeHealthRecord(health node.Health) deployment.NodeHealth { diff --git a/services/core/internal/execution/deployment_fixture_test.go b/services/core/internal/execution/deployment_fixture_test.go index 1cb8c27b5..ad9a16417 100644 --- a/services/core/internal/execution/deployment_fixture_test.go +++ b/services/core/internal/execution/deployment_fixture_test.go @@ -51,15 +51,6 @@ func unitDeploymentService(t *testing.T) *deployment.Service { return service } -// unusedPreparation is the preparer of a test that submits no sandbox -// selection; preparing one fails the test. -func unusedPreparation(t *testing.T) RuntimeDeploymentPreparer { - return func(context.Context, deployment.Setup) (PreparedRuntimeDeployment, error) { - t.Error("the test prepared a sandbox selection it did not submit") - return PreparedRuntimeDeployment{}, errors.New("unexpected sandbox selection preparation") - } -} - // deploymentOperations builds the deployment service on storage and reader and // the execution operations on execution. func deploymentOperations(t *testing.T, storage deployment.Storage, reader deployment.Reader, execution deployment.ExecutionStorage) (*deployment.Service, *deployment.ExecutionOperations) { diff --git a/services/core/internal/execution/dispatcher.go b/services/core/internal/execution/dispatcher.go index d9b6304d0..6c0cf9c4b 100644 --- a/services/core/internal/execution/dispatcher.go +++ b/services/core/internal/execution/dispatcher.go @@ -14,6 +14,7 @@ import ( "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/deployment" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/modelconfiguration" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/runtimegateway" + "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sessions" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/vaults" ) @@ -52,9 +53,13 @@ type Dispatcher struct { // durable Serve authority or an assignment's attach authority ends, and // before it destroys the resource or sends the release. It is required. Links *relay.Relay - // ManagedRuntimes provisions hosted Environments on the sandbox deployment. - // It is required. - ManagedRuntimes *RuntimeProvider + // Providers and NodeProviders construct direct adapters and fixed node proxies. + // Both are required; the runtime manager owns deployment setup and routing. + Providers ProviderRegistry + NodeProviders NodeProviders + ProviderPaths sandbox.ProcessPaths + InstallationID string + SandboxLink string // MaxConcurrentExecutions bounds work admitted by this Core execution owner. // Zero uses DefaultExecutionConcurrency. It is independent of sandbox capacity. MaxConcurrentExecutions int diff --git a/services/core/cmd/server/managed_generations_test.go b/services/core/internal/execution/managed_generations_test.go similarity index 91% rename from services/core/cmd/server/managed_generations_test.go rename to services/core/internal/execution/managed_generations_test.go index 7e5855f6b..9c3132683 100644 --- a/services/core/cmd/server/managed_generations_test.go +++ b/services/core/internal/execution/managed_generations_test.go @@ -1,4 +1,4 @@ -package main +package execution import ( "context" @@ -54,9 +54,9 @@ print(json.dumps({'Version':1,'Info':info})) value.Configuration = &key return value, nil } - setup := &managedSetup{processPaths: paths, registry: providers.Builtin(), deployment: &fakeDeploymentSetups{t: t, allocationSetup: allocation}, installationID: id} + setup := &runtimeManager{processPaths: paths, providers: providers.Builtin(), setups: &fakeDeploymentSetups{t: t, allocationSetup: allocation}, setupInstallationID: id} // A facade retained by a generation-one lifecycle still reads current credentials. - router := &generationRouter{setup: setup} + router := sandbox.NewGenerationRouter(e2b.Operations(), setup.directProvider) ctx, cancel := context.WithTimeout(t.Context(), 5*time.Second) defer cancel() if _, err := router.GetInfo(ctx, ref); err != nil { @@ -150,19 +150,19 @@ print(json.dumps(result)) committed := current setups := &fakeDeploymentSetups{t: t, setup: committedSetup(&committed), withCredential: credentialService(t), generationPage: func(context.Context, int64) ([]deployment.Setup, error) { return nil, nil }} - allocations := &fakeGenerationAllocations{t: t, credentialAllocations: func(context.Context, string) ([]deployment.Allocation, error) { return nil, nil }} - s := &managedSetup{processPaths: paths, registry: providers.Builtin(), installationID: id, deployment: setups, allocations: allocations} - loaded, err := s.load(t.Context()) + allocations := &strictDeploymentReader{t: t, credentialAllocations: func(context.Context, string) ([]deployment.Allocation, error) { return nil, nil }} + s := &runtimeManager{processPaths: paths, providers: providers.Builtin(), setupInstallationID: id, setups: setups, deploymentReader: allocations} + loaded, err := s.loadDeployment(t.Context()) if err != nil { t.Fatal(err) } next := current next.Configuration = &e2b.DeploymentConfiguration{APIKey: tc.candidateKey, Template: tc.candidateTemplate + build} - candidate, err := s.prepare(t.Context(), next) + candidate, err := s.prepareDeployment(t.Context(), next) if err != nil { t.Fatal(err) } - err = candidate.VerifyCredential(t.Context()) + err = s.verifyCredential(t.Context(), candidate.Setup) var reset *deployment.ResetRequiredError if tc.reset { if !errors.As(err, &reset) || errors.Is(err, sandbox.ErrCredentialRejected) || errors.Is(err, sandbox.ErrCredentialOwnership) { diff --git a/services/core/cmd/server/managed_setup_preflight_test.go b/services/core/internal/execution/managed_setup_preflight_test.go similarity index 78% rename from services/core/cmd/server/managed_setup_preflight_test.go rename to services/core/internal/execution/managed_setup_preflight_test.go index dc977e41a..79dfefa9d 100644 --- a/services/core/cmd/server/managed_setup_preflight_test.go +++ b/services/core/internal/execution/managed_setup_preflight_test.go @@ -1,4 +1,4 @@ -package main +package execution import ( "context" @@ -10,7 +10,6 @@ import ( "time" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/deployment" - "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/execution" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox/e2b" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox/providers" @@ -28,14 +27,14 @@ func TestE2BRejectedSpecificationHasSafeActionableDiagnostic(t *testing.T) { } paths := testProviderPaths(t, helper, state) id := uuid.NewString() - s := &managedSetup{processPaths: paths, registry: providers.Builtin(), installationID: id} + s := &runtimeManager{processPaths: paths, providers: providers.Builtin(), setupInstallationID: id} selection := deployment.Setup{InstallationID: id, Provider: "e2b", Mode: "direct", UsesCredential: true, Specification: sandbox.DeploymentSpec{Resources: sandbox.Resources{CPUs: 3, MemoryMiB: 3072}}, Configuration: &e2b.DeploymentConfiguration{APIKey: "synthetic-private-key", Template: "runtime:" + uuid.NewString()}} - _, err := s.prepare(t.Context(), selection) + _, err := s.prepareDeployment(t.Context(), selection) if !errors.Is(err, sandbox.ErrConfigurationSelection) || strings.Contains(err.Error(), "synthetic-private-key") || s.selected.Load() != nil { t.Fatal("rejected candidate lost its safe diagnostic or was published", err) } - s.deployment = &fakeDeploymentSetups{t: t, setup: committedSetup(&selection)} - if restored, err := s.load(t.Context()); err != nil || restored == nil || restored.Provider == nil { + s.setups = &fakeDeploymentSetups{t: t, setup: committedSetup(&selection)} + if restored, err := s.loadDeployment(t.Context()); err != nil || restored == nil || restored.Provider == nil { t.Fatal("template rejection prevented loading committed resource ownership", err) } } @@ -69,12 +68,12 @@ else: Specification: sandbox.DeploymentSpec{Resources: sandbox.Resources{CPUs: 2, MemoryMiB: 2048}}, Configuration: &e2b.DeploymentConfiguration{APIKey: "synthetic-private-key", Template: "runtime:" + uuid.NewString()}} allocation := func(context.Context, sandbox.Reference) (deployment.Setup, error) { return selection, nil } - s := &managedSetup{processPaths: paths, registry: providers.Builtin(), installationID: id, - deployment: &fakeDeploymentSetups{t: t, setup: committedSetup(&selection), allocationSetup: allocation}} - if _, err := s.prepare(t.Context(), selection); err == nil || s.selected.Load() != nil { + s := &runtimeManager{processPaths: paths, providers: providers.Builtin(), setupInstallationID: id, + setups: &fakeDeploymentSetups{t: t, setup: committedSetup(&selection), allocationSetup: allocation}} + if _, err := s.prepareDeployment(t.Context(), selection); err == nil || s.selected.Load() != nil { t.Fatal("invalid new template selection was published", err) } - loaded, err := s.load(t.Context()) + loaded, err := s.loadDeployment(t.Context()) if err != nil || loaded == nil { t.Fatal("committed provider became inaccessible", err) } @@ -107,9 +106,9 @@ func TestE2BCandidateAdoptsTemplateBuildForOmittedResources(t *testing.T) { } paths := testProviderPaths(t, helper, state) id := uuid.NewString() - s := &managedSetup{processPaths: paths, registry: providers.Builtin(), installationID: id, deployment: &fakeDeploymentSetups{t: t}} + s := &runtimeManager{processPaths: paths, providers: providers.Builtin(), setupInstallationID: id, setups: &fakeDeploymentSetups{t: t}} selection := deployment.Setup{InstallationID: id, Provider: "e2b", Mode: "direct", UsesCredential: true, Configuration: &e2b.DeploymentConfiguration{APIKey: "synthetic-private-key", Template: "runtime:" + uuid.NewString()}} - candidate, err := s.prepare(t.Context(), selection) + candidate, err := s.prepareDeployment(t.Context(), selection) disk := int32(24063) if err != nil || candidate.Selection.Configuration.(*e2b.DeploymentConfiguration).TemplateBuild == nil || candidate.Selection.Configuration.(*e2b.DeploymentConfiguration).TemplateBuild.CPUs != 4 || candidate.Selection.Configuration.(*e2b.DeploymentConfiguration).TemplateBuild.MemoryMiB != 4096 || *candidate.Selection.Configuration.(*e2b.DeploymentConfiguration).TemplateBuild.RootDiskMiB != disk || candidate.Selection.Configuration.(*e2b.DeploymentConfiguration).TemplateBuild.Status != "ready" { @@ -117,11 +116,11 @@ func TestE2BCandidateAdoptsTemplateBuildForOmittedResources(t *testing.T) { } // The published candidate enforces the adopted resources. selection.Specification.Resources = sandbox.Resources{CPUs: 4, MemoryMiB: 4096} - provider, err := s.provider(selection) + _, err = s.provider(selection) if err != nil { t.Fatal(err) } - if _, err := provider.(*e2b.Provider).ValidateDeployment(t.Context()); err != nil { + if _, err := s.providers.DiscoverSelection(t.Context(), s.direct(selection)); err != nil { t.Fatal(err) } logged, err := os.ReadFile(requests) @@ -142,21 +141,9 @@ func TestInitialE2BPublicTemplateOutsideTeamIsRejected(t *testing.T) { } paths := testProviderPaths(t, helper, state) id := uuid.NewString() - s := &managedSetup{processPaths: paths, registry: providers.Builtin(), installationID: id, deployment: &fakeDeploymentSetups{t: t}} + s := &runtimeManager{processPaths: paths, providers: providers.Builtin(), setupInstallationID: id, setups: &fakeDeploymentSetups{t: t}} selection := deployment.Setup{InstallationID: id, Provider: "e2b", Mode: "direct", UsesCredential: true, Configuration: &e2b.DeploymentConfiguration{APIKey: "synthetic-team-a", Template: "public-team-b:" + uuid.NewString()}} - if _, err := s.prepare(t.Context(), selection); !errors.Is(err, sandbox.ErrCredentialOwnership) || s.selected.Load() != nil { + if _, err := s.prepareDeployment(t.Context(), selection); !errors.Is(err, sandbox.ErrCredentialOwnership) || s.selected.Load() != nil { t.Fatal("public readability accepted as team ownership", err) } } - -func TestManagedSetupRoutesProviderWithoutCredentialRequirement(t *testing.T) { - s := &managedSetup{} - config := &execution.RuntimeProvider{ProviderKind: "docker"} - candidate, err := s.routeGenerations( - execution.PreparedRuntimeDeployment{Config: config}, - deployment.Setup{Provider: "docker", Mode: "nodes"}, - ) - if err != nil || candidate.Config != config || candidate.FenceCredential != nil || candidate.VerifyCredential != nil { - t.Fatalf("explicit no-credential provider required credential routing: %v", err) - } -} diff --git a/services/core/cmd/server/managed_setup_test.go b/services/core/internal/execution/managed_setup_test.go similarity index 67% rename from services/core/cmd/server/managed_setup_test.go rename to services/core/internal/execution/managed_setup_test.go index 3967268f9..7681ffc5a 100644 --- a/services/core/cmd/server/managed_setup_test.go +++ b/services/core/internal/execution/managed_setup_test.go @@ -1,4 +1,4 @@ -package main +package execution import ( "context" @@ -11,31 +11,17 @@ import ( "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/deployment/placement" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/persistence/postgres/deploymentpg" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/persistence/postgres/pgtest" - "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/processconfig" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/runtimeobs" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox/docker" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox/e2b" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox/microsandbox" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox/providers" - "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/execution" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox/node" "github.com/google/uuid" ) -func TestWebSetupCreatesManagerWithoutLocalProvider(t *testing.T) { - origin, err := deployment.NewPublicOrigin("https://core.example") - if err != nil { - t.Fatal(err) - } - m := configureManagedNodes(nil, nil, providers.Builtin(), processconfig.Config{InstallationID: uuid.NewString(), PublicOrigin: origin}, func(context.Context) error { return nil }) - defer m.hub.Close() - if m.setup == nil || m.hub == nil || m.runtime == nil || m.runtime.Provider != nil || m.setup.sandboxLink != "wss://core.example/api/v1/sandbox-link" { - t.Fatal("zero-node setup unexpectedly instantiated local compute or omitted management") - } -} - // fakeDeploymentSetups is a strict deploymentSetups: a call without a set // function fails the test. type fakeDeploymentSetups struct { @@ -78,19 +64,6 @@ func (f *fakeDeploymentSetups) AllocationGeneration(ctx context.Context, ref san return f.allocationGeneration(ctx, ref) } -// fakeGenerationAllocations is a strict generationAllocations. -type fakeGenerationAllocations struct { - t testing.TB - credentialAllocations func(context.Context, string) ([]deployment.Allocation, error) -} - -func (f *fakeGenerationAllocations) CredentialAllocations(ctx context.Context, after string) ([]deployment.Allocation, error) { - if f.credentialAllocations == nil { - return nil, unexpectedCall(f.t, "CredentialAllocations") - } - return f.credentialAllocations(ctx, after) -} - // unexpectedCall fails the test from any goroutine and returns the error the // caller propagates. func unexpectedCall(t testing.TB, method string) error { @@ -122,29 +95,29 @@ func TestManagedSetupNeverReusesAnotherGenerationOrUnverifiedState(t *testing.T) value := deployment.Setup{InstallationID: "installation", Provider: "docker", Mode: "nodes", Generation: 1} var loadErr error setups := &fakeDeploymentSetups{t: t, setup: func(context.Context) (deployment.Setup, error) { return value, loadErr }} - s := &managedSetup{registry: providers.Builtin(), deployment: setups, allocations: &fakeGenerationAllocations{t: t}, installationID: "installation"} - cached := &execution.RuntimeProvider{InstallationID: "installation", ProviderKind: "docker", Generation: 1} - s.publish(cached) - if got, err := s.load(t.Context()); err != nil || got != cached { + s := &runtimeManager{providers: providers.Builtin(), setups: setups, deploymentReader: &strictDeploymentReader{t: t}, setupInstallationID: "installation"} + cached := &RuntimeProvider{InstallationID: "installation", ProviderKind: "docker", Generation: 1} + s.publishSelection(cached.Generation, cached) + if got, err := s.loadDeployment(t.Context()); err != nil || got != cached { t.Fatal("matching immutable selection was not reused") } loadErr = errors.New("database unavailable") - if _, err := s.load(t.Context()); err == nil || errors.Is(err, execution.ErrExecutionUnavailable) { + if _, err := s.loadDeployment(t.Context()); err == nil || errors.Is(err, ErrExecutionUnavailable) { t.Fatal("stale cached selection hid storage failure", err) } loadErr = deployment.ErrCredentialUnreadable - if _, err := s.load(t.Context()); !errors.Is(err, execution.ErrExecutionUnavailable) || !errors.Is(err, deployment.ErrCredentialUnreadable) { + if _, err := s.loadDeployment(t.Context()); !errors.Is(err, ErrExecutionUnavailable) || !errors.Is(err, deployment.ErrCredentialUnreadable) { t.Fatal("an unreadable credential must block execution without stopping Core", err) } loadErr = nil value.Generation = 2 // No Hub is installed; a changed generation must construct again and fail. - if _, err := s.load(t.Context()); !errors.Is(err, execution.ErrExecutionUnavailable) { + if _, err := s.loadDeployment(t.Context()); !errors.Is(err, ErrExecutionUnavailable) { t.Fatal("changed provider availability must block execution without losing recovery", err) } value.InstallationID = "other-installation" value.Generation = 1 - if _, err := s.load(t.Context()); err == nil { + if _, err := s.loadDeployment(t.Context()); err == nil { t.Fatal("cache ignored installation identity") } } @@ -153,9 +126,9 @@ func TestMissingE2BHelperReportsProviderUnavailable(t *testing.T) { id := uuid.NewString() committed := deployment.Setup{InstallationID: id, Provider: "e2b", Mode: "direct", Generation: 1, UsesCredential: true, Configuration: &e2b.DeploymentConfiguration{APIKey: "synthetic-key", Template: "runtime:" + uuid.NewString()}} - s := &managedSetup{processPaths: sandbox.ProcessPaths{ArtifactRoot: t.TempDir(), StateRoot: t.TempDir()}, registry: providers.Builtin(), installationID: id, - deployment: &fakeDeploymentSetups{t: t, setup: committedSetup(&committed)}} - if _, err := s.load(t.Context()); !errors.Is(err, execution.ErrExecutionUnavailable) { + s := &runtimeManager{processPaths: sandbox.ProcessPaths{ArtifactRoot: t.TempDir(), StateRoot: t.TempDir()}, providers: providers.Builtin(), setupInstallationID: id, + setups: &fakeDeploymentSetups{t: t, setup: committedSetup(&committed)}} + if _, err := s.loadDeployment(t.Context()); !errors.Is(err, ErrExecutionUnavailable) { t.Fatal("missing local helper must leave administrative recovery available", err) } if s.selected.Load() != nil { @@ -167,10 +140,10 @@ func TestManagedSetupPreparesWithoutPublishing(t *testing.T) { id := uuid.NewString() hub := node.NewHub(node.HubOptions{}) defer hub.Close() - s := &managedSetup{registry: providers.Builtin(), installationID: id, hub: hub, deployment: &fakeDeploymentSetups{t: t}, allocations: &fakeGenerationAllocations{t: t}, sandboxLink: "wss://core.example/api/v1/sandbox-link"} - previous := &execution.RuntimeProvider{InstallationID: id, Generation: 1, ProviderKind: "docker"} - s.publish(previous) - candidate, err := s.prepare(t.Context(), deployment.Setup{InstallationID: id, Provider: "microsandbox", Mode: "nodes", Operations: microsandbox.Operations(), Suspension: &deployment.Suspension{IdleSeconds: 300, RetentionSeconds: 86400}}) + s := &runtimeManager{providers: providers.Builtin(), setupInstallationID: id, nodeProviders: hub, setups: &fakeDeploymentSetups{t: t}, deploymentReader: &strictDeploymentReader{t: t}, sandboxLink: "wss://core.example/api/v1/sandbox-link"} + previous := &RuntimeProvider{InstallationID: id, Generation: 1, ProviderKind: "docker"} + s.publishSelection(previous.Generation, previous) + candidate, err := s.prepareDeployment(t.Context(), deployment.Setup{InstallationID: id, Provider: "microsandbox", Mode: "nodes", Operations: microsandbox.Operations(), Suspension: &deployment.Suspension{IdleSeconds: 300, RetentionSeconds: 86400}}) if err != nil { t.Fatal(err) } @@ -179,7 +152,7 @@ func TestManagedSetupPreparesWithoutPublishing(t *testing.T) { } committed := *candidate.Config committed.Generation = 2 - candidate.Publish(&committed) + s.publishSelection(committed.Generation, &committed) if got := s.selected.Load(); got.Generation != 2 || got.Config.ProviderKind != "microsandbox" { t.Fatal("commit did not publish the validated selection") } @@ -187,12 +160,12 @@ func TestManagedSetupPreparesWithoutPublishing(t *testing.T) { func TestManagedSetupRejectedCandidateRetainsSelection(t *testing.T) { id := uuid.NewString() - s := &managedSetup{processPaths: sandbox.ProcessPaths{ArtifactRoot: t.TempDir(), StateRoot: t.TempDir()}, registry: providers.Builtin(), installationID: id} - previous := &execution.RuntimeProvider{InstallationID: id, Generation: 1, ProviderKind: "docker"} - s.publish(previous) - _, err := s.prepare(t.Context(), deployment.Setup{InstallationID: id, Provider: "e2b", Mode: "direct", UsesCredential: true, + s := &runtimeManager{processPaths: sandbox.ProcessPaths{ArtifactRoot: t.TempDir(), StateRoot: t.TempDir()}, providers: providers.Builtin(), setupInstallationID: id} + previous := &RuntimeProvider{InstallationID: id, Generation: 1, ProviderKind: "docker"} + s.publishSelection(previous.Generation, previous) + _, err := s.prepareDeployment(t.Context(), deployment.Setup{InstallationID: id, Provider: "e2b", Mode: "direct", UsesCredential: true, Configuration: &e2b.DeploymentConfiguration{APIKey: "synthetic-key", Template: "runtime:" + uuid.NewString()}}) - if !errors.Is(err, execution.ErrExecutionUnavailable) || s.selected.Load().Config != previous { + if !errors.Is(err, ErrExecutionUnavailable) || s.selected.Load().Config != previous { t.Fatal("rejected candidate lost the previous selection", err) } } @@ -210,17 +183,17 @@ func TestManagedSetupResetTombstoneRejectsDelayedProviderLoad(t *testing.T) { return deployment.Setup{}, ctx.Err() } } - s := &managedSetup{registry: providers.Builtin(), installationID: id, deployment: &fakeDeploymentSetups{t: t, setup: delayed}} + s := &runtimeManager{providers: providers.Builtin(), setupInstallationID: id, setups: &fakeDeploymentSetups{t: t, setup: delayed}} done := make(chan error, 1) go func() { - provider, err := s.load(t.Context()) + provider, err := s.loadDeployment(t.Context()) if err == nil && provider != nil { err = errors.New("old provider survived reset") } done <- err }() <-entered - s.publishUnconfigured(2) + s.publishSelection(2, nil) close(release) if err := <-done; err != nil { t.Fatal(err) @@ -228,12 +201,12 @@ func TestManagedSetupResetTombstoneRejectsDelayedProviderLoad(t *testing.T) { if s.selected.Load().Generation != 2 || s.selected.Load().Config != nil { t.Fatal("empty publication lost its generation") } - s.publish(&execution.RuntimeProvider{Generation: 1, ProviderKind: "docker"}) + s.publishSelection(1, &RuntimeProvider{Generation: 1, ProviderKind: "docker"}) if s.selected.Load().Config != nil { t.Fatal("late old publication resurrected provider") } - next := &execution.RuntimeProvider{Generation: 3, ProviderKind: "microsandbox"} - s.publish(next) + next := &RuntimeProvider{Generation: 3, ProviderKind: "microsandbox"} + s.publishSelection(next.Generation, next) if s.selected.Load().Config != next { t.Fatal("reset blocked subsequent configuration") } @@ -261,21 +234,21 @@ func testProviderPaths(t *testing.T, helper, state string) sandbox.ProcessPaths func TestManagedObservationSourceKeepsSelectionAcrossReconfiguration(t *testing.T) { value := deployment.Setup{InstallationID: "installation", Generation: 1} - setup := &managedSetup{registry: providers.Builtin(), deployment: &fakeDeploymentSetups{t: t, setup: committedSetup(&value)}, installationID: "installation"} - if source, kind, err := setup.observationSource(t.Context()); source != nil || kind != "" || !errors.Is(err, runtimeobs.ErrUnavailable) { + setup := &runtimeManager{providers: providers.Builtin(), setups: &fakeDeploymentSetups{t: t, setup: committedSetup(&value)}, setupInstallationID: "installation"} + if source, kind, err := (&Worker{runtimes: setup}).ObservationSource(t.Context()); source != nil || kind != "" || !errors.Is(err, runtimeobs.ErrUnavailable) { t.Fatal("unconfigured setup did not return typed unavailability", source, err) } first := &docker.Provider{} value.Provider, value.Mode, value.Generation = "docker", "nodes", 2 - setup.publish(&execution.RuntimeProvider{Generation: 2, ProviderKind: "docker", Provider: first}) - source, kind, err := setup.observationSource(t.Context()) + setup.publishSelection(2, &RuntimeProvider{Generation: 2, ProviderKind: "docker", Provider: first}) + source, kind, err := (&Worker{runtimes: setup}).ObservationSource(t.Context()) if err != nil || source != first || kind != "docker" { t.Fatal(source, kind, err) } next := µsandbox.Provider{} value.Provider, value.Mode, value.Generation = "microsandbox", "nodes", 3 - setup.publish(&execution.RuntimeProvider{Generation: 3, ProviderKind: "microsandbox", Provider: next}) - selected, kind, err := setup.observationSource(t.Context()) + setup.publishSelection(3, &RuntimeProvider{Generation: 3, ProviderKind: "microsandbox", Provider: next}) + selected, kind, err := (&Worker{runtimes: setup}).ObservationSource(t.Context()) if err != nil || selected != next || kind != "microsandbox" { t.Fatal(selected, kind, err) } diff --git a/services/core/internal/execution/owner_test.go b/services/core/internal/execution/owner_test.go index e03ccbaea..76cc96435 100644 --- a/services/core/internal/execution/owner_test.go +++ b/services/core/internal/execution/owner_test.go @@ -66,12 +66,6 @@ func unusedSessions(t *testing.T) (*sessions.Service, sessions.Reader) { return service, struct{ sessions.Reader }{} } -// unusedRuntimes is a runtime provider for a new installation whose -// deployment no operator has configured. -func unusedRuntimes(t *testing.T) *RuntimeProvider { - return NewDeferredRuntimeProvider(uuid.NewString(), func(context.Context) (*RuntimeProvider, error) { return nil, nil }, unusedPreparation(t)) -} - // unusedObserver fails the test on any observation. The Workers it serves run // no Turn. type unusedObserver struct{ t *testing.T } @@ -134,20 +128,20 @@ func TestStartWorkerFailureClosesLeaseOnce(t *testing.T) { "missing Session operations": func(t *testing.T, lease *closeCountingLease) error { _, deployments, deploymentReader := resetManager(t) service, reader := unusedSessions(t) - _, err := StartWorker(canceled, &Dispatcher{Credentials: &recordingCredentials{}, Observer: unusedObserver{t}, Deployment: deployments, DeploymentReader: deploymentReader, Sessions: service, SessionsReader: reader, ManagedRuntimes: unusedRuntimes(t)}, Owner{Lease: lease}) + _, err := StartWorker(canceled, &Dispatcher{Credentials: &recordingCredentials{}, Observer: unusedObserver{t}, Deployment: deployments, DeploymentReader: deploymentReader, Sessions: service, SessionsReader: reader, Providers: &fakeProviderRegistry{t: t}, NodeProviders: fixtureNodeProviders{t: t}, InstallationID: uuid.NewString()}, Owner{Lease: lease}) return err }, "missing deployment": func(t *testing.T, lease *closeCountingLease) error { owner, deployments, deploymentReader := resetManager(t) service, reader := unusedSessions(t) - _, err := StartWorker(canceled, &Dispatcher{Credentials: &recordingCredentials{}, Observer: unusedObserver{t}, Deployment: deployments, DeploymentReader: deploymentReader, Sessions: service, SessionsReader: reader, ManagedRuntimes: unusedRuntimes(t)}, Owner{Lease: lease, Sessions: owner.Sessions}) + _, err := StartWorker(canceled, &Dispatcher{Credentials: &recordingCredentials{}, Observer: unusedObserver{t}, Deployment: deployments, DeploymentReader: deploymentReader, Sessions: service, SessionsReader: reader, Providers: &fakeProviderRegistry{t: t}, NodeProviders: fixtureNodeProviders{t: t}, InstallationID: uuid.NewString()}, Owner{Lease: lease, Sessions: owner.Sessions}) return err }, "deployment claim": func(t *testing.T, lease *closeCountingLease) error { owner, deployments, deploymentReader := resetManager(t) lease.inner = owner.Lease service, reader := unusedSessions(t) - dispatcher := &Dispatcher{Registry: runtimegateway.NewRegistry(), Credentials: &recordingCredentials{}, Observer: unusedObserver{t}, Deployment: deployments, DeploymentReader: deploymentReader, Sessions: service, SessionsReader: reader, Links: relay.New(nil), ManagedRuntimes: unusedRuntimes(t)} + dispatcher := &Dispatcher{Registry: runtimegateway.NewRegistry(), Credentials: &recordingCredentials{}, Observer: unusedObserver{t}, Deployment: deployments, DeploymentReader: deploymentReader, Sessions: service, SessionsReader: reader, Links: relay.New(nil), Providers: &fakeProviderRegistry{t: t}, NodeProviders: fixtureNodeProviders{t: t}, InstallationID: uuid.NewString()} _, err := StartWorker(canceled, dispatcher, Owner{Lease: lease, Deployment: owner.Deployment, Sessions: owner.Sessions}) if ping := owner.Lease.CheckOwnership(t.Context()); !errors.Is(ping, pgunit.ErrLeaseClosed) { t.Error("failed start kept the database lease", ping) @@ -172,7 +166,6 @@ func TestStartWorkerChecksDeploymentAfterItsDependencies(t *testing.T) { owner, deployments, deploymentReader := resetManager(t) credentials, observer := &recordingCredentials{}, unusedObserver{t} service, reader := unusedSessions(t) - runtimes := unusedRuntimes(t) bound := Owner{Sessions: owner.Sessions} for _, test := range []struct { name string @@ -187,9 +180,9 @@ func TestStartWorkerChecksDeploymentAfterItsDependencies(t *testing.T) { {"missing Session service", Dispatcher{Credentials: credentials, Observer: observer, Deployment: deployments, DeploymentReader: deploymentReader}, bound, "execution worker requires the Session service"}, {"missing Session reader", Dispatcher{Credentials: credentials, Observer: observer, Deployment: deployments, DeploymentReader: deploymentReader, Sessions: service}, bound, "execution worker requires the Session reader"}, {"missing runtime provider", Dispatcher{Credentials: credentials, Observer: observer, Deployment: deployments, DeploymentReader: deploymentReader, Sessions: service, SessionsReader: reader}, bound, "execution worker requires the sandbox runtime provider"}, - {"missing Session operations", Dispatcher{Credentials: credentials, Observer: observer, Deployment: deployments, DeploymentReader: deploymentReader, Sessions: service, SessionsReader: reader, ManagedRuntimes: runtimes}, Owner{}, "execution requires the Session execution operations"}, - {"missing deployment", Dispatcher{Credentials: credentials, Observer: observer, Deployment: deployments, DeploymentReader: deploymentReader, Sessions: service, SessionsReader: reader, ManagedRuntimes: runtimes}, bound, "execution worker requires the deployment execution operations"}, - {"missing Link relay", Dispatcher{Credentials: credentials, Observer: observer, Deployment: deployments, DeploymentReader: deploymentReader, Sessions: service, SessionsReader: reader, ManagedRuntimes: runtimes}, Owner{Sessions: owner.Sessions, Deployment: owner.Deployment}, "execution worker requires the Link relay"}, + {"missing Session operations", Dispatcher{Credentials: credentials, Observer: observer, Deployment: deployments, DeploymentReader: deploymentReader, Sessions: service, SessionsReader: reader, Providers: &fakeProviderRegistry{t: t}}, Owner{}, "execution requires the Session execution operations"}, + {"missing deployment", Dispatcher{Credentials: credentials, Observer: observer, Deployment: deployments, DeploymentReader: deploymentReader, Sessions: service, SessionsReader: reader, Providers: &fakeProviderRegistry{t: t}}, bound, "execution worker requires the deployment execution operations"}, + {"missing Link relay", Dispatcher{Credentials: credentials, Observer: observer, Deployment: deployments, DeploymentReader: deploymentReader, Sessions: service, SessionsReader: reader, Providers: &fakeProviderRegistry{t: t}}, Owner{Sessions: owner.Sessions, Deployment: owner.Deployment}, "execution worker requires the Link relay"}, } { t.Run(test.name, func(t *testing.T) { lease := &closeCountingLease{t: t} @@ -208,7 +201,7 @@ func TestWorkerRunClosesLeaseAfterDrain(t *testing.T) { lease := &closeCountingLease{t: t, inner: owner.Lease} // The Worker's first reconciliation scans the Session work. reader, service := testSessions(t, pool, pgtest.CredentialKey(t)) - dispatcher := &Dispatcher{Registry: runtimegateway.NewRegistry(), Credentials: &recordingCredentials{}, Observer: unusedObserver{t}, Deployment: deployments, DeploymentReader: deploymentReader, Sessions: service, SessionsReader: reader, Links: relay.New(nil), ManagedRuntimes: unusedRuntimes(t)} + dispatcher := &Dispatcher{Registry: runtimegateway.NewRegistry(), Credentials: &recordingCredentials{}, Observer: unusedObserver{t}, Deployment: deployments, DeploymentReader: deploymentReader, Sessions: service, SessionsReader: reader, Links: relay.New(nil), Providers: &fakeProviderRegistry{t: t}, NodeProviders: fixtureNodeProviders{t: t}, InstallationID: uuid.NewString()} worker, err := StartWorker(t.Context(), dispatcher, Owner{Lease: lease, Deployment: owner.Deployment, Sessions: owner.Sessions}) if err != nil { t.Fatal(err) diff --git a/services/core/internal/execution/provider_fixture_test.go b/services/core/internal/execution/provider_fixture_test.go new file mode 100644 index 000000000..45098943e --- /dev/null +++ b/services/core/internal/execution/provider_fixture_test.go @@ -0,0 +1,85 @@ +package execution + +import ( + "context" + "encoding/json" + "testing" + + "github.com/MiniMax-AI/OpenAgentCore/internal/sandboxlink/relay" + "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/deployment" + "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/providercontract" + "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/runtimegateway" + "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox" + "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox/providers" +) + +type fakeProviderRegistry struct { + t testing.TB + lookup func(string) (providers.Adapter, error) + build func(sandbox.DirectConfig) (sandbox.SandboxProvider, error) + discover func(context.Context, sandbox.DirectConfig) (sandbox.Selection, error) + verify func(context.Context, sandbox.DirectConfig, []sandbox.Reference) error +} + +func (f *fakeProviderRegistry) Lookup(kind string) (providers.Adapter, error) { + if f.lookup == nil { + return providers.Adapter{}, unexpectedCall(f.t, "Lookup") + } + return f.lookup(kind) +} +func (f *fakeProviderRegistry) BuildDirect(c sandbox.DirectConfig) (sandbox.SandboxProvider, error) { + if f.build == nil { + return nil, unexpectedCall(f.t, "BuildDirect") + } + return f.build(c) +} +func (f *fakeProviderRegistry) DiscoverSelection(ctx context.Context, c sandbox.DirectConfig) (sandbox.Selection, error) { + if f.discover == nil { + return sandbox.Selection{}, unexpectedCall(f.t, "DiscoverSelection") + } + return f.discover(ctx, c) +} +func (f *fakeProviderRegistry) VerifyCredential(ctx context.Context, c sandbox.DirectConfig, refs []sandbox.Reference) error { + if f.verify == nil { + return unexpectedCall(f.t, "VerifyCredential") + } + return f.verify(ctx, c, refs) +} +func (f *fakeProviderRegistry) DiscoverConfiguration(context.Context, string, sandbox.ConfigurationDiscoveryInput, sandbox.ProcessPaths) (json.RawMessage, error) { + return nil, unexpectedCall(f.t, "DiscoverConfiguration") +} + +type fixtureNodeProviders struct { + t testing.TB + provider sandbox.SandboxProvider +} + +func (f fixtureNodeProviders) Proxy(string, providercontract.Operations, uint64) sandbox.SandboxProvider { + if f.provider == nil { + unexpectedCall(f.t, "Proxy") + } + return f.provider +} + +func testManager(t *testing.T, owner Owner, deployments *deployment.Service, reader deployment.Reader, id string, p sandbox.SandboxProvider) (*runtimeManager, error) { + t.Helper() + registry := &fakeProviderRegistry{t: t, lookup: providers.Builtin().Lookup, build: func(sandbox.DirectConfig) (sandbox.SandboxProvider, error) { + if p == nil { + return nil, unexpectedCall(t, "BuildDirect") + } + return p, nil + }, discover: func(_ context.Context, c sandbox.DirectConfig) (sandbox.Selection, error) { return c.Selection, nil }} + d := &Dispatcher{Deployment: deployments, DeploymentReader: reader, Registry: runtimegateway.NewRegistry(), Links: relay.New(nil), InstallationID: id, SandboxLink: "wss://core.example/api/v1/sandbox-link", Providers: registry, NodeProviders: fixtureNodeProviders{t: t, provider: p}} + return newRuntimeManager(owner, d, nil) +} + +// selectTestProvider supplies a committed immutable setup and its adapter. +// Loading still uses the manager's real construction and publication path. +func selectTestProvider(t *testing.T, m *runtimeManager, config *RuntimeProvider) { + t.Helper() + setup := deployment.Setup{InstallationID: config.InstallationID, Provider: config.ProviderKind, Mode: config.Mode, Generation: config.Generation, BackendFingerprint: config.BackendFingerprint, Operations: config.Provider.ProviderOperations()} + m.setups = &fakeDeploymentSetups{t: t, setup: committedSetup(&setup), allocationSetup: func(context.Context, sandbox.Reference) (deployment.Setup, error) { return setup, nil }} + m.providers = &fakeProviderRegistry{t: t, build: func(sandbox.DirectConfig) (sandbox.SandboxProvider, error) { return config.Provider, nil }, lookup: providers.Builtin().Lookup} + m.nodeProviders = fixtureNodeProviders{t: t, provider: config.Provider} + m.sandboxLink = config.SandboxLink +} diff --git a/services/core/internal/execution/runtime_lifecycle.go b/services/core/internal/execution/runtime_lifecycle.go index 5d05c3a3a..290510a1c 100644 --- a/services/core/internal/execution/runtime_lifecycle.go +++ b/services/core/internal/execution/runtime_lifecycle.go @@ -24,18 +24,14 @@ import ( // RuntimeProvider binds one deployment to one sandbox installation. // BackendFingerprint identifies its namespace independently of mutable sizing. type RuntimeProvider struct { - // PublishUnconfigured updates the shared observation cache after reset commit. - PublishUnconfigured func(uint64) - Generation uint64 - Mode string - loadDeployment func(context.Context) (*RuntimeProvider, error) - prepareDeployment RuntimeDeploymentPreparer - ProviderKind string - SandboxLink string - InstallationID string - BackendFingerprint string - Provider sandbox.SandboxProvider - Suspension *RuntimeSuspensionPolicy + Generation uint64 + Mode string + ProviderKind string + SandboxLink string + InstallationID string + BackendFingerprint string + Provider sandbox.SandboxProvider + Suspension *RuntimeSuspensionPolicy } type runtimeLifecycle struct { @@ -62,16 +58,15 @@ type runtimeLifecycle struct { wakeHints chan struct{} } -func newRuntimeManager(owner Owner, deployments *deployment.Service, deploymentReader deployment.Reader, sessionReader sessions.Reader, registry *runtimegateway.Registry, links *relay.Relay, connections *environmentConnections, config *RuntimeProvider) (*runtimeManager, error) { - if config == nil { - return nil, sandbox.ErrInvalid - } - id, err := uuid.Parse(config.InstallationID) - if err != nil || id == uuid.Nil || id.String() != config.InstallationID || config.loadDeployment == nil || config.prepareDeployment == nil || registry == nil { +func newRuntimeManager(owner Owner, d *Dispatcher, connections *environmentConnections) (*runtimeManager, error) { + id, err := uuid.Parse(d.InstallationID) + if err != nil || id == uuid.Nil || id.String() != d.InstallationID || d.Providers == nil || d.NodeProviders == nil || d.Registry == nil { return nil, sandbox.ErrInvalid } ctx, stop := context.WithCancel(context.Background()) - return &runtimeManager{sessions: sessionReader, sessionExecution: owner.Sessions, deployment: owner.Deployment, deploymentService: deployments, deploymentReader: deploymentReader, lease: owner.Lease, registry: registry, links: links, connections: connections, setupInstallationID: config.InstallationID, loadDeployment: config.loadDeployment, prepareDeployment: config.prepareDeployment, publishUnconfigured: config.PublishUnconfigured, setupGate: make(chan struct{}, 1), mutationGate: make(chan struct{}, 1), ctx: ctx, cancel: stop, nodes: make(map[string]*runtimeNode), failed: make(chan error, 1), inventory: make(chan struct{}, 1)}, nil + return &runtimeManager{sessions: d.SessionsReader, sessionExecution: owner.Sessions, deployment: owner.Deployment, deploymentService: d.Deployment, deploymentReader: d.DeploymentReader, lease: owner.Lease, registry: d.Registry, links: d.Links, connections: connections, setupInstallationID: d.InstallationID, + setups: d.Deployment, providers: d.Providers, nodeProviders: d.NodeProviders, processPaths: d.ProviderPaths, sandboxLink: d.SandboxLink, + setupGate: make(chan struct{}, 1), mutationGate: make(chan struct{}, 1), ctx: ctx, cancel: stop, nodes: make(map[string]*runtimeNode), failed: make(chan error, 1), inventory: make(chan struct{}, 1)}, nil } func validatedRuntimeProvider(config *RuntimeProvider, registry *runtimegateway.Registry) (RuntimeProvider, error) { diff --git a/services/core/internal/execution/runtime_manager.go b/services/core/internal/execution/runtime_manager.go index 0e5914747..b22aed845 100644 --- a/services/core/internal/execution/runtime_manager.go +++ b/services/core/internal/execution/runtime_manager.go @@ -5,11 +5,13 @@ import ( "errors" "fmt" "sync" + "sync/atomic" "time" "github.com/MiniMax-AI/OpenAgentCore/internal/sandboxlink/relay" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/deployment" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/runtimegateway" + "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sessions" ) @@ -29,9 +31,13 @@ type runtimeManager struct { connections *environmentConnections config RuntimeProvider setupInstallationID string - loadDeployment func(context.Context) (*RuntimeProvider, error) - prepareDeployment RuntimeDeploymentPreparer - publishUnconfigured func(uint64) + setups deploymentSetups + providers ProviderRegistry + nodeProviders NodeProviders + processPaths sandbox.ProcessPaths + sandboxLink string + selected atomic.Pointer[managedSelection] + providerCalls sandbox.CallFence resetCursor string resetRequestedAt time.Time setupGate chan struct{} diff --git a/services/core/internal/execution/runtime_manager_test.go b/services/core/internal/execution/runtime_manager_test.go index a01528e43..d717c2cb3 100644 --- a/services/core/internal/execution/runtime_manager_test.go +++ b/services/core/internal/execution/runtime_manager_test.go @@ -23,7 +23,7 @@ func (heldLease) Close(context.Context) error { return nil } func testRuntimeManager(t *testing.T) *runtimeManager { t.Helper() ctx, cancel := context.WithCancel(t.Context()) - m := &runtimeManager{lease: heldLease{}, config: RuntimeProvider{ProviderKind: "docker", Mode: "nodes", Provider: &drainFixtureProvider{}}, loadDeployment: func(context.Context) (*RuntimeProvider, error) { return nil, nil }, ctx: ctx, cancel: cancel, nodes: make(map[string]*runtimeNode), failed: make(chan error, 1), inventory: make(chan struct{}, 1)} + m := &runtimeManager{lease: heldLease{}, config: RuntimeProvider{ProviderKind: "docker", Mode: "nodes", Provider: &drainFixtureProvider{}}, ctx: ctx, cancel: cancel, nodes: make(map[string]*runtimeNode), failed: make(chan error, 1), inventory: make(chan struct{}, 1)} t.Cleanup(func() { m.stop(); m.drain() }) return m } diff --git a/services/core/cmd/server/managed_setup.go b/services/core/internal/execution/sandbox_configuration.go similarity index 52% rename from services/core/cmd/server/managed_setup.go rename to services/core/internal/execution/sandbox_configuration.go index cf8cfaa80..916271e37 100644 --- a/services/core/cmd/server/managed_setup.go +++ b/services/core/internal/execution/sandbox_configuration.go @@ -1,41 +1,20 @@ -package main +package execution import ( "context" "encoding/json" "errors" "fmt" - "sync/atomic" "time" "github.com/MiniMax-AI/OpenAgentCore/internal/obs/log" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/deployment" - "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/execution" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/providercontract" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/runtimeobs" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox" - "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox/node" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox/providers" ) -// managedSetup publishes one immutable selection to execution, bootstrap and -// observation. The database owns the selection; this cache is never a writer. -type managedSetup struct { - processPaths sandbox.ProcessPaths - // registry builds the selected direct provider and discovers configuration; - // the deployment setup reports what the registration declares. - registry *providers.Registry - deployment deploymentSetups - allocations generationAllocations - hub *node.Hub - installationID string - // sandboxLink is the Link every sandbox Serves on, empty when the origin - // has no Link. - sandboxLink string - selected atomic.Pointer[managedSelection] - providerCalls sandbox.CallFence -} - // deploymentSetups reads the committed deployment setup and its retained // generations. *deployment.Service implements it. type deploymentSetups interface { @@ -48,26 +27,37 @@ type deploymentSetups interface { AllocationGeneration(context.Context, sandbox.Reference) (string, uint64, error) } -// generationAllocations pages the unreleased allocations whose generations -// still hold a credential. deployment.Reader implements it. -type generationAllocations interface { - CredentialAllocations(context.Context, string) ([]deployment.Allocation, error) +// ProviderRegistry is the setup and direct-construction dependency of execution. +// The composition root supplies the registered providers; support stays declared +// by their ConfigurationAdapter and SandboxProvider contracts. +type ProviderRegistry interface { + Lookup(string) (providers.Adapter, error) + BuildDirect(sandbox.DirectConfig) (sandbox.SandboxProvider, error) + DiscoverConfiguration(context.Context, string, sandbox.ConfigurationDiscoveryInput, sandbox.ProcessPaths) (json.RawMessage, error) + DiscoverSelection(context.Context, sandbox.DirectConfig) (sandbox.Selection, error) + VerifyCredential(context.Context, sandbox.DirectConfig, []sandbox.Reference) error +} + +// NodeProviders binds allocation-owned node and generation identities to the +// node transport. The server supplies its node.Hub. +type NodeProviders interface { + Proxy(string, providercontract.Operations, uint64) sandbox.SandboxProvider } // DiscoverConfiguration asks a Provider which configuration values its // credential can use, with this installation's process paths. -func (s *managedSetup) DiscoverConfiguration(ctx context.Context, provider string, input sandbox.ConfigurationDiscoveryInput) (json.RawMessage, error) { - return s.registry.DiscoverConfiguration(ctx, provider, input, s.processPaths) +func (w *Worker) DiscoverConfiguration(ctx context.Context, provider string, input sandbox.ConfigurationDiscoveryInput) (json.RawMessage, error) { + return w.runtimes.providers.DiscoverConfiguration(ctx, provider, input, w.runtimes.processPaths) } // Empty selections retain their generation so a delayed provider load cannot // republish a backend retired by reset. type managedSelection struct { Generation uint64 - Config *execution.RuntimeProvider + Config *RuntimeProvider } -func (s *managedSetup) publishSelection(generation uint64, config *execution.RuntimeProvider) *execution.RuntimeProvider { +func (s *runtimeManager) publishSelection(generation uint64, config *RuntimeProvider) *RuntimeProvider { next := &managedSelection{Generation: generation, Config: config} for { current := s.selected.Load() @@ -79,22 +69,18 @@ func (s *managedSetup) publishSelection(generation uint64, config *execution.Run } } } -func (s *managedSetup) publish(config *execution.RuntimeProvider) { - s.publishSelection(config.Generation, config) -} -func (s *managedSetup) publishUnconfigured(generation uint64) { s.publishSelection(generation, nil) } -func (s *managedSetup) load(ctx context.Context) (*execution.RuntimeProvider, error) { - setup, err := s.deployment.Setup(ctx) +func (s *runtimeManager) loadDeployment(ctx context.Context) (*RuntimeProvider, error) { + setup, err := s.setups.Setup(ctx) if errors.Is(err, deployment.ErrCredentialUnreadable) { // A replaced credential key blocks hosted execution, not Core. log.Warn(ctx, "Hosted provider credential is unreadable; administrator recovery remains available", "error", err) - return nil, fmt.Errorf("%w: %w", execution.ErrExecutionUnavailable, err) + return nil, fmt.Errorf("%w: %w", ErrExecutionUnavailable, err) } if err != nil { return nil, err } - if setup.InstallationID != s.installationID { + if setup.InstallationID != s.setupInstallationID { return nil, errors.New("sandbox installation does not match setup") } if setup.Provider == "" { @@ -104,65 +90,63 @@ func (s *managedSetup) load(ctx context.Context) (*execution.RuntimeProvider, er return selected.Config, nil } candidate, err := s.configuration(setup) - if err == nil { - candidate, err = s.routeGenerations(candidate, setup) - } if err != nil { log.Warn(ctx, "Hosted provider is unavailable; administrator recovery remains available", "provider", setup.Provider, "error", err) - return nil, fmt.Errorf("%w: %v", execution.ErrExecutionUnavailable, err) + return nil, fmt.Errorf("%w: %v", ErrExecutionUnavailable, err) } return s.publishSelection(setup.Generation, candidate.Config), nil } // prepare validates a setup the deployment prepared for a selection, which has // already rejected a provider whose guests cannot reach the public URL. -func (s *managedSetup) prepare(ctx context.Context, setup deployment.Setup) (execution.PreparedRuntimeDeployment, error) { +func (s *runtimeManager) prepareDeployment(ctx context.Context, setup deployment.Setup) (preparedRuntimeDeployment, error) { candidate, err := s.configuration(setup) if err != nil { - return execution.PreparedRuntimeDeployment{}, err + return preparedRuntimeDeployment{}, err } - adapter, err := s.registry.Lookup(setup.Provider) + adapter, err := s.providers.Lookup(setup.Provider) if err != nil { - return execution.PreparedRuntimeDeployment{}, err + return preparedRuntimeDeployment{}, err } direct := s.direct(setup) selection := direct.Selection if adapter.Configuration.Requirements().SelectionDiscovery.State == providercontract.Supported { - if selection, err = s.registry.DiscoverSelection(ctx, direct); err != nil { - return execution.PreparedRuntimeDeployment{}, err + if selection, err = s.providers.DiscoverSelection(ctx, direct); err != nil { + return preparedRuntimeDeployment{}, err } setup.Specification, setup.Configuration = selection.DeploymentSpec, selection.Configuration if candidate, err = s.configuration(setup); err != nil { - return execution.PreparedRuntimeDeployment{}, err + return preparedRuntimeDeployment{}, err } } - candidate.Selection = &selection - return s.routeGenerations(candidate, setup) + candidate.Selection = selection + candidate.Setup = setup + return candidate, nil } // Loading an already committed selection must retain provider access to its // owned resources, even when a new-template validation would now fail. -func (s *managedSetup) configuration(setup deployment.Setup) (execution.PreparedRuntimeDeployment, error) { - if setup.InstallationID != s.installationID { - return execution.PreparedRuntimeDeployment{}, errors.New("sandbox installation does not match setup") +func (s *runtimeManager) configuration(setup deployment.Setup) (preparedRuntimeDeployment, error) { + if setup.InstallationID != s.setupInstallationID { + return preparedRuntimeDeployment{}, errors.New("sandbox installation does not match setup") } provider, err := s.provider(setup) if err != nil { - return execution.PreparedRuntimeDeployment{}, fmt.Errorf("%w: %v", execution.ErrExecutionUnavailable, err) + return preparedRuntimeDeployment{}, fmt.Errorf("%w: %v", ErrExecutionUnavailable, err) } - selected := &execution.RuntimeProvider{InstallationID: setup.InstallationID, ProviderKind: setup.Provider, Generation: setup.Generation, Mode: setup.Mode, + selected := &RuntimeProvider{InstallationID: setup.InstallationID, ProviderKind: setup.Provider, Generation: setup.Generation, Mode: setup.Mode, SandboxLink: s.sandboxLink, BackendFingerprint: setup.BackendFingerprint, Provider: provider} if setup.Suspension != nil { - selected.Suspension = &execution.RuntimeSuspensionPolicy{IdleTimeout: time.Duration(setup.Suspension.IdleSeconds) * time.Second, + selected.Suspension = &RuntimeSuspensionPolicy{IdleTimeout: time.Duration(setup.Suspension.IdleSeconds) * time.Second, Retention: time.Duration(setup.Suspension.RetentionSeconds) * time.Second} } - return execution.PreparedRuntimeDeployment{Config: selected, Publish: s.publish}, nil + return preparedRuntimeDeployment{Config: selected, Setup: setup}, nil } -// observationSource returns the selected Provider and its registered kind for +// ObservationSource returns the selected Provider and its registered kind for // Runtime observation. -func (s *managedSetup) observationSource(ctx context.Context) (runtimeobs.Source, string, error) { - selected, err := s.load(ctx) +func (w *Worker) ObservationSource(ctx context.Context) (runtimeobs.Source, string, error) { + selected, err := w.runtimes.loadDeployment(ctx) if err != nil { return nil, "", err } @@ -174,17 +158,31 @@ func (s *managedSetup) observationSource(ctx context.Context) (runtimeobs.Source // provider builds the setup's provider. The setup carries the mode and // declared operations that deployment read from the provider's registration. -func (s *managedSetup) provider(setup deployment.Setup) (sandbox.SandboxProvider, error) { +func (s *runtimeManager) provider(setup deployment.Setup) (sandbox.SandboxProvider, error) { if setup.Mode == "nodes" { - if s.hub == nil { + if s.nodeProviders == nil { return nil, errors.New("sandbox node transport is unavailable") } - return s.hub.GenerationProvider(setup.Operations, s.deployment.AllocationGeneration), nil + return sandbox.NewGenerationRouter(setup.Operations, func(ctx context.Context, ref sandbox.Reference) (sandbox.SandboxProvider, func(), error) { + id, generation, err := s.setups.AllocationGeneration(ctx, ref) + if err != nil { + return nil, nil, err + } + return s.nodeProviders.Proxy(id, setup.Operations, generation), func() {}, nil + }), nil + } + provider, err := s.providers.BuildDirect(s.direct(setup)) + if err != nil { + return nil, err + } + routed := sandbox.NewGenerationRouter(provider.ProviderOperations(), s.directProvider) + if err := sandbox.ValidateProvider(routed); err != nil { + return nil, err } - return s.registry.BuildDirect(s.direct(setup)) + return routed, nil } // direct is the setup's input to direct-mode construction and setup operations. -func (s *managedSetup) direct(setup deployment.Setup) sandbox.DirectConfig { +func (s *runtimeManager) direct(setup deployment.Setup) sandbox.DirectConfig { return sandbox.DirectConfig{ProcessPaths: s.processPaths, InstallationID: setup.InstallationID, Selection: sandbox.Selection{Provider: setup.Provider, DeploymentSpec: setup.Specification, Configuration: setup.Configuration}, Fence: &s.providerCalls} } diff --git a/services/core/internal/execution/sandbox_credentials.go b/services/core/internal/execution/sandbox_credentials.go new file mode 100644 index 000000000..70187c4e8 --- /dev/null +++ b/services/core/internal/execution/sandbox_credentials.go @@ -0,0 +1,109 @@ +package execution + +import ( + "context" + "errors" + "time" + + "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/deployment" + "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox" +) + +// Retained facades read allocation-owned specifications with the current +// credential inside the same fence as the native call. +func (s *runtimeManager) directProvider(ctx context.Context, r sandbox.Reference) (sandbox.SandboxProvider, func(), error) { + release, err := s.providerCalls.Enter(ctx) + if err != nil { + return nil, nil, err + } + setup, err := s.setups.AllocationSetup(ctx, r) + if err != nil { + release() + return nil, nil, err + } + provider, err := s.providers.BuildDirect(s.direct(setup)) + if err != nil { + release() + return nil, nil, err + } + return provider, release, nil +} +func (s *runtimeManager) verifyCredential(ctx context.Context, setup deployment.Setup) error { + ctx, cancel := context.WithTimeout(ctx, 30*time.Second) + defer cancel() + // Preserve the committed credential until its ownership anchor is verified. + // Public template readability cannot establish which team owns a deployment. + verify := func(value deployment.Setup, refs []sandbox.Reference) error { + value.InstallationID = setup.InstallationID + return s.providers.VerifyCredential(ctx, s.direct(value), refs) + } + current, err := s.setups.Setup(ctx) + if err != nil { + return err + } + if current.Provider != setup.Provider { + return &deployment.ResetRequiredError{CurrentProvider: current.Provider, RequestedProvider: setup.Provider} + } + if err := verify(current, nil); err != nil { + if errors.Is(err, sandbox.ErrCredentialRejected) || errors.Is(err, sandbox.ErrCredentialOwnership) { + // A revoked legacy key or a public template outside its team cannot + // anchor ownership. This says nothing about the candidate key's validity. + return &deployment.ResetRequiredError{CurrentProvider: setup.Provider, RequestedProvider: setup.Provider} + } + return err + } + withCandidateKey := func(value deployment.Setup, refs []sandbox.Reference) error { + value, err := s.setups.WithCredential(value, setup) + if err != nil { + return err + } + return verify(value, refs) + } + if err := withCandidateKey(current, nil); err != nil { + return err + } + if err := verify(setup, nil); err != nil { + return err + } + generations := map[uint64]deployment.Setup{current.Generation: current} + for after := int64(-1); ; { + page, err := s.setups.GenerationPage(ctx, after) + if err != nil { + return err + } + for _, g := range page { + generations[g.Generation] = g + if err := withCandidateKey(g, nil); err != nil { + return err + } + after = int64(g.Generation) + } + if len(page) < 32 { + break + } + } + for after := ""; ; { + page, err := s.deploymentReader.CredentialAllocations(ctx, after) + if err != nil { + return err + } + refsByGeneration := make(map[uint64][]sandbox.Reference) + for _, a := range page { + refsByGeneration[a.DeploymentGeneration] = append(refsByGeneration[a.DeploymentGeneration], sandbox.Reference{TenantID: a.TenantID, EnvironmentID: a.EnvironmentID, AllocationID: a.ID}) + after = a.ID + } + for generation, refs := range refsByGeneration { + owner, ok := generations[generation] + if !ok { + return sandbox.ErrConfigurationUnconfirmed + } + if err := withCandidateKey(owner, refs); err != nil { + return err + } + } + if len(page) < 32 { + break + } + } + return nil +} diff --git a/services/core/internal/execution/sandbox_deployment_drain_test.go b/services/core/internal/execution/sandbox_deployment_drain_test.go index 4b1920ec3..29410852e 100644 --- a/services/core/internal/execution/sandbox_deployment_drain_test.go +++ b/services/core/internal/execution/sandbox_deployment_drain_test.go @@ -10,11 +10,9 @@ import ( "testing" "time" - "github.com/MiniMax-AI/OpenAgentCore/internal/sandboxlink/relay" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/deployment" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/persistence/postgres/pgtest" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/persistence/postgres/pgunit" - "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/runtimegateway" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox/docker" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox/node" "github.com/google/uuid" @@ -94,11 +92,12 @@ func testLifecycleCancellationPreservesLease(t *testing.T, mode string) { defer hub.Close() id := uuid.NewString() configuration := &RuntimeProvider{InstallationID: id, ProviderKind: "docker", Mode: "nodes", Generation: 1, SandboxLink: "wss://core.example/api/v1/sandbox-link", BackendFingerprint: strings.Repeat("a", 64), Provider: hub.Proxy(uuid.NewString(), docker.Operations(), 1)} - m, err := newRuntimeManager(owner, deployments, reader, nil, runtimegateway.NewRegistry(), relay.New(nil), nil, NewDeferredRuntimeProvider(id, func(context.Context) (*RuntimeProvider, error) { return configuration, nil }, unusedPreparation(t))) + m, err := testManager(t, owner, deployments, reader, id, configuration.Provider) if err != nil { t.Fatal(err) } defer func() { unblock(); m.stop(); m.drain() }() + selectTestProvider(t, m, configuration) if _, err := m.ensureDeployment(t.Context()); err != nil { t.Fatal(err) } @@ -232,11 +231,12 @@ func TestSandboxDeploymentDrainFailureCannotReactivate(t *testing.T) { defer hub.Close() id := uuid.NewString() configuration := &RuntimeProvider{InstallationID: id, ProviderKind: "docker", Mode: "nodes", Generation: 1, SandboxLink: "wss://core.example/api/v1/sandbox-link", BackendFingerprint: strings.Repeat("a", 64), Provider: hub.Proxy(uuid.NewString(), docker.Operations(), 1)} - m, err := newRuntimeManager(owner, deployments, reader, nil, runtimegateway.NewRegistry(), relay.New(nil), nil, NewDeferredRuntimeProvider(id, func(context.Context) (*RuntimeProvider, error) { return configuration, nil }, unusedPreparation(t))) + m, err := testManager(t, owner, deployments, reader, id, configuration.Provider) if err != nil { t.Fatal(err) } defer func() { m.stop(); m.drain() }() + selectTestProvider(t, m, configuration) if _, err := m.ensureDeployment(t.Context()); err != nil { t.Fatal(err) } diff --git a/services/core/internal/execution/sandbox_deployment_setup.go b/services/core/internal/execution/sandbox_deployment_setup.go index 6ccf10692..59be763f6 100644 --- a/services/core/internal/execution/sandbox_deployment_setup.go +++ b/services/core/internal/execution/sandbox_deployment_setup.go @@ -8,25 +8,11 @@ import ( "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox" ) -// PreparedRuntimeDeployment has completed provider validation without publishing -// a selection. Publish must only update in-memory state and must not fail. -// Selection is the resolved typed configuration returned by preparation. -type PreparedRuntimeDeployment struct { - Config *RuntimeProvider - Publish func(*RuntimeProvider) - Selection *sandbox.Selection - VerifyCredential func(context.Context) error - FenceCredential func(context.Context) (func(), error) -} - -type RuntimeDeploymentPreparer func(context.Context, deployment.Setup) (PreparedRuntimeDeployment, error) - -// NewDeferredRuntimeProvider enables Web setup for one fixed installation. The -// loader returns nil until selection, then the committed immutable generation; -// prepare validates each new selection before it is stored. Replacement is -// serialized by the deployment mutation gate and drain flow. -func NewDeferredRuntimeProvider(installationID string, load func(context.Context) (*RuntimeProvider, error), prepare RuntimeDeploymentPreparer) *RuntimeProvider { - return &RuntimeProvider{InstallationID: installationID, loadDeployment: load, prepareDeployment: prepare} +// preparedRuntimeDeployment is validated but not yet committed or published. +type preparedRuntimeDeployment struct { + Config *RuntimeProvider + Selection sandbox.Selection + Setup deployment.Setup } func (w *Worker) InitializeSandboxDeployment(ctx context.Context, input sandbox.Selection) (deployment.View, error) { @@ -53,7 +39,7 @@ func (w *Worker) InitializeSandboxDeployment(ctx context.Context, input sandbox. return deployment.View{}, err } } - result, err := m.deployment.Initialize(ctx, m.setupInstallationID, *candidate.Selection) + result, err := m.deployment.Initialize(ctx, m.setupInstallationID, candidate.Selection) if err != nil { return deployment.View{}, err } @@ -91,7 +77,7 @@ func (m *runtimeManager) ensureDeployment(parent context.Context) (bool, error) if err != nil || config == nil { return false, err } - if config.InstallationID != m.setupInstallationID || config.loadDeployment != nil { + if config.InstallationID != m.setupInstallationID { return false, sandbox.ErrInvalid } copied, err := validatedRuntimeProvider(config, m.registry) @@ -112,37 +98,34 @@ func (m *runtimeManager) ensureDeployment(parent context.Context) (bool, error) // Preparation is outside the manager mutex and all database transactions. A // rejected candidate cannot retire the current generation or its node lanes. -func (m *runtimeManager) prepareCandidate(ctx context.Context, input sandbox.Selection) (PreparedRuntimeDeployment, error) { +func (m *runtimeManager) prepareCandidate(ctx context.Context, input sandbox.Selection) (preparedRuntimeDeployment, error) { setup, err := m.deploymentService.SetupForSelection(m.setupInstallationID, input) if err != nil { - return PreparedRuntimeDeployment{}, err + return preparedRuntimeDeployment{}, err } candidate, err := m.prepareDeployment(ctx, setup) if err != nil { - return PreparedRuntimeDeployment{}, err + return preparedRuntimeDeployment{}, err } config := candidate.Config - if config == nil || config.InstallationID != setup.InstallationID || config.ProviderKind != setup.Provider || config.Mode != setup.Mode || config.BackendFingerprint != setup.BackendFingerprint || config.loadDeployment != nil || config.prepareDeployment != nil { - return PreparedRuntimeDeployment{}, sandbox.ErrInvalid + if config == nil || config.InstallationID != setup.InstallationID || config.ProviderKind != setup.Provider || config.Mode != setup.Mode || config.BackendFingerprint != setup.BackendFingerprint { + return preparedRuntimeDeployment{}, sandbox.ErrInvalid } copied, err := validatedRuntimeProvider(config, m.registry) if err != nil { - return PreparedRuntimeDeployment{}, err + return preparedRuntimeDeployment{}, err } if err := ctx.Err(); err != nil { - return PreparedRuntimeDeployment{}, err + return preparedRuntimeDeployment{}, err } m.mu.Lock() closed := m.closed m.mu.Unlock() if closed { - return PreparedRuntimeDeployment{}, ErrExecutionUnavailable - } - if candidate.Selection == nil { - candidate.Selection = &input + return preparedRuntimeDeployment{}, ErrExecutionUnavailable } if candidate.Selection.Provider != input.Provider { - return PreparedRuntimeDeployment{}, sandbox.ErrInvalid + return preparedRuntimeDeployment{}, sandbox.ErrInvalid } candidate.Selection.ExpectedGeneration = input.ExpectedGeneration candidate.Config = &copied @@ -151,7 +134,7 @@ func (m *runtimeManager) prepareCandidate(ctx context.Context, input sandbox.Sel // The deployment commit is the point of no return. Publishing a validated candidate // is infallible, including when shutdown or request cancellation follows commit. -func (m *runtimeManager) publishDeployment(candidate PreparedRuntimeDeployment, committed deployment.View) { +func (m *runtimeManager) publishDeployment(candidate preparedRuntimeDeployment, committed deployment.View) { m.mu.Lock() defer m.mu.Unlock() config := *candidate.Config @@ -160,9 +143,7 @@ func (m *runtimeManager) publishDeployment(candidate PreparedRuntimeDeployment, m.nodes = make(map[string]*runtimeNode) } m.config = config - if candidate.Publish != nil { - candidate.Publish(&config) - } + m.publishSelection(config.Generation, &config) m.switching = false m.switchDrained = nil } diff --git a/services/core/internal/execution/sandbox_deployment_setup_test.go b/services/core/internal/execution/sandbox_deployment_setup_test.go index 39f107eb8..41b759ecd 100644 --- a/services/core/internal/execution/sandbox_deployment_setup_test.go +++ b/services/core/internal/execution/sandbox_deployment_setup_test.go @@ -9,9 +9,7 @@ import ( "testing" "time" - "github.com/MiniMax-AI/OpenAgentCore/internal/sandboxlink/relay" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/deployment" - "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/runtimegateway" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox/docker" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox/e2b" @@ -26,13 +24,14 @@ func TestDeferredSandboxDeploymentLoadsOnceBeforeNodeCreation(t *testing.T) { var selected atomic.Bool var loads atomic.Int32 configuration := &RuntimeProvider{InstallationID: id, ProviderKind: "docker", Mode: "nodes", SandboxLink: "wss://core.example/api/v1/sandbox-link", BackendFingerprint: strings.Repeat("a", 64), Provider: hub.Proxy(uuid.NewString(), docker.Operations(), 1)} - m, err := newRuntimeManager(Owner{Lease: heldLease{}}, nil, nil, nil, runtimegateway.NewRegistry(), relay.New(nil), nil, NewDeferredRuntimeProvider(id, func(context.Context) (*RuntimeProvider, error) { + m, err := testManager(t, Owner{Lease: heldLease{}}, nil, nil, id, configuration.Provider) + m.setups = &fakeDeploymentSetups{t: t, setup: func(context.Context) (deployment.Setup, error) { loads.Add(1) if !selected.Load() { - return nil, nil + return deployment.Setup{InstallationID: id}, nil } - return configuration, nil - }, unusedPreparation(t))) + return deployment.Setup{InstallationID: id, Provider: "docker", Mode: "nodes", Generation: 1, BackendFingerprint: configuration.BackendFingerprint, Operations: docker.Operations()}, nil + }} if err != nil { t.Fatal(err) } @@ -71,11 +70,12 @@ func TestDeferredSandboxDeploymentLoadsOnceBeforeNodeCreation(t *testing.T) { func TestDeferredSandboxDeploymentShutdownCancelsLoad(t *testing.T) { entered := make(chan struct{}) - m, err := newRuntimeManager(Owner{Lease: heldLease{}}, nil, nil, nil, runtimegateway.NewRegistry(), relay.New(nil), nil, NewDeferredRuntimeProvider(uuid.NewString(), func(ctx context.Context) (*RuntimeProvider, error) { + m, err := testManager(t, Owner{Lease: heldLease{}}, nil, nil, uuid.NewString(), nil) + m.setups = &fakeDeploymentSetups{t: t, setup: func(ctx context.Context) (deployment.Setup, error) { close(entered) <-ctx.Done() - return nil, ctx.Err() - }, unusedPreparation(t))) + return deployment.Setup{}, ctx.Err() + }} if err != nil { t.Fatal(err) } @@ -96,12 +96,13 @@ func TestDeferredSandboxProviderFailureKeepsRecoveryAvailable(t *testing.T) { available := false loadErr := ErrExecutionUnavailable configuration := &RuntimeProvider{InstallationID: id, ProviderKind: "docker", Mode: "nodes", SandboxLink: "wss://core.example/api/v1/sandbox-link", BackendFingerprint: strings.Repeat("a", 64), Provider: hub.Proxy(uuid.NewString(), docker.Operations(), 1)} - m, err := newRuntimeManager(Owner{Lease: heldLease{}}, nil, nil, nil, runtimegateway.NewRegistry(), relay.New(nil), nil, NewDeferredRuntimeProvider(id, func(context.Context) (*RuntimeProvider, error) { + m, err := testManager(t, Owner{Lease: heldLease{}}, nil, nil, id, configuration.Provider) + m.setups = &fakeDeploymentSetups{t: t, setup: func(context.Context) (deployment.Setup, error) { if !available { - return nil, loadErr + return deployment.Setup{}, loadErr } - return configuration, nil - }, unusedPreparation(t))) + return deployment.Setup{InstallationID: id, Provider: "docker", Mode: "nodes", Generation: 1, BackendFingerprint: configuration.BackendFingerprint, Operations: docker.Operations()}, nil + }} if err != nil { t.Fatal(err) } @@ -128,10 +129,9 @@ func TestRejectedSandboxCandidatePreservesActiveGeneration(t *testing.T) { id := uuid.NewString() config := &RuntimeProvider{InstallationID: id, ProviderKind: "docker", Mode: "nodes", Generation: 1, SandboxLink: "wss://core.example/api/v1/sandbox-link", BackendFingerprint: strings.Repeat("a", 64), Provider: hub.Proxy(uuid.NewString(), docker.Operations(), 1)} rejected := errors.New("candidate provider unavailable") - m, err := newRuntimeManager(Owner{Lease: heldLease{}}, unitDeploymentService(t), nil, nil, runtimegateway.NewRegistry(), relay.New(nil), nil, NewDeferredRuntimeProvider(id, func(context.Context) (*RuntimeProvider, error) { return config, nil }, - func(context.Context, deployment.Setup) (PreparedRuntimeDeployment, error) { - return PreparedRuntimeDeployment{}, rejected - })) + m, err := testManager(t, Owner{Lease: heldLease{}}, unitDeploymentService(t), nil, id, config.Provider) + selectTestProvider(t, m, config) + m.providers = &fakeProviderRegistry{t: t, build: func(sandbox.DirectConfig) (sandbox.SandboxProvider, error) { return nil, rejected }} if err != nil { t.Fatal(err) } @@ -144,10 +144,10 @@ func TestRejectedSandboxCandidatePreservesActiveGeneration(t *testing.T) { t.Fatal(err) } input := sandbox.Selection{Provider: "e2b", Configuration: &e2b.DeploymentConfiguration{APIKey: "fixture-key", Template: "runtime:" + uuid.NewString()}, DeploymentSpec: sandbox.DeploymentSpec{Resources: sandbox.Resources{CPUs: 2, MemoryMiB: 1024}}} - if _, err := m.prepareCandidate(t.Context(), input); !errors.Is(err, rejected) { + if _, err := m.prepareCandidate(t.Context(), input); err == nil || !strings.Contains(err.Error(), rejected.Error()) { t.Fatal("candidate rejection was lost", err) } - if m.config.Generation != 1 || m.config.Provider != config.Provider || m.switching || old.lifecycle.ctx.Err() != nil { + if m.config.Generation != 1 || m.config.Provider != old.lifecycle.config.Provider || m.switching || old.lifecycle.ctx.Err() != nil { t.Fatal("rejected candidate replaced or drained the active configuration") } _, finish, err := m.enter(t.Context()) @@ -160,12 +160,12 @@ func TestRejectedSandboxCandidatePreservesActiveGeneration(t *testing.T) { func TestSandboxCandidateValidationDoesNotHoldManagerLock(t *testing.T) { id := uuid.NewString() entered, release := make(chan struct{}), make(chan struct{}) - m, err := newRuntimeManager(Owner{Lease: heldLease{}}, unitDeploymentService(t), nil, nil, runtimegateway.NewRegistry(), relay.New(nil), nil, NewDeferredRuntimeProvider(id, func(context.Context) (*RuntimeProvider, error) { return nil, nil }, - func(context.Context, deployment.Setup) (PreparedRuntimeDeployment, error) { - close(entered) - <-release - return PreparedRuntimeDeployment{}, errors.New("rejected") - })) + m, err := testManager(t, Owner{Lease: heldLease{}}, unitDeploymentService(t), nil, id, nil) + m.providers = &fakeProviderRegistry{t: t, build: func(sandbox.DirectConfig) (sandbox.SandboxProvider, error) { + close(entered) + <-release + return nil, errors.New("rejected") + }} if err != nil { t.Fatal(err) } @@ -193,7 +193,7 @@ func TestCommittedSandboxCandidatePublishesAfterShutdown(t *testing.T) { hub := node.NewHub(node.HubOptions{}) defer hub.Close() id := uuid.NewString() - m, err := newRuntimeManager(Owner{Lease: heldLease{}}, nil, nil, nil, runtimegateway.NewRegistry(), relay.New(nil), nil, NewDeferredRuntimeProvider(id, func(context.Context) (*RuntimeProvider, error) { return nil, nil }, unusedPreparation(t))) + m, err := testManager(t, Owner{Lease: heldLease{}}, nil, nil, id, nil) if err != nil { t.Fatal(err) } @@ -201,12 +201,11 @@ func TestCommittedSandboxCandidatePublishesAfterShutdown(t *testing.T) { t.Fatal(err) } config := &RuntimeProvider{InstallationID: id, ProviderKind: "e2b", Mode: "direct", SandboxLink: "wss://core.example/api/v1/sandbox-link", BackendFingerprint: strings.Repeat("b", 64), Provider: hub.Proxy(uuid.NewString(), docker.Operations(), 1)} - var published *RuntimeProvider - candidate := PreparedRuntimeDeployment{Config: config, Publish: func(value *RuntimeProvider) { published = value }} + candidate := preparedRuntimeDeployment{Config: config} m.stop() m.publishDeployment(candidate, deployment.View{InstallationID: id, Generation: 2, Mode: "direct", Provider: "e2b", Reset: &deployment.Reset{}}) m.drain() - if m.config.Generation != 2 || published == nil || published.Generation != 2 || m.switching { + if published := m.selected.Load(); m.config.Generation != 2 || published == nil || published.Generation != 2 || m.switching { t.Fatal("committed candidate was lost during shutdown") } if _, _, err := m.enter(t.Context()); !errors.Is(err, ErrExecutionUnavailable) { diff --git a/services/core/internal/execution/sandbox_deployment_switch.go b/services/core/internal/execution/sandbox_deployment_switch.go index 0dddd2923..6ed80bf45 100644 --- a/services/core/internal/execution/sandbox_deployment_switch.go +++ b/services/core/internal/execution/sandbox_deployment_switch.go @@ -115,7 +115,7 @@ func (m *runtimeManager) activateDeployment(ctx context.Context, expected deploy m.publishEmptyDeployment(expected.InstallationID, expected.Generation) return nil } - if config == nil || config.InstallationID != expected.InstallationID || config.Generation != expected.Generation || config.Mode != expected.Mode || config.ProviderKind != expected.Provider || config.loadDeployment != nil { + if config == nil || config.InstallationID != expected.InstallationID || config.Generation != expected.Generation || config.Mode != expected.Mode || config.ProviderKind != expected.Provider { return sandbox.ErrInvalid } copied, err := validatedRuntimeProvider(config, m.registry) @@ -155,28 +155,36 @@ func (w *Worker) UpdateSandboxDeployment(ctx context.Context, input sandbox.Sele if err != nil { return deployment.View{}, err } - if input.HasCredential() || candidate.VerifyCredential != nil { - if candidate.VerifyCredential == nil || candidate.FenceCredential == nil { + if input.HasCredential() || candidate.Setup.UsesCredential { + if !candidate.Setup.UsesCredential { return deployment.View{}, ErrExecutionUnavailable } - if err := candidate.VerifyCredential(ctx); err != nil { + adapter, err := m.providers.Lookup(candidate.Setup.Provider) + if err != nil { + return deployment.View{}, err + } + if err := adapter.Configuration.Requirements().CredentialVerification.Check("VerifyCredential"); err != nil { + return deployment.View{}, err + } + if err := m.verifyCredential(ctx, candidate.Setup); err != nil { return deployment.View{}, err } if input.ReplacesCredential() { fenceCtx, cancel := context.WithTimeout(ctx, 30*time.Second) defer cancel() - release, err := candidate.FenceCredential(fenceCtx) + release, err := m.providerCalls.Fence(fenceCtx) if err != nil { - return deployment.View{}, err + return deployment.View{}, sandbox.ErrConfigurationUnconfirmed } defer release() // The final scan includes allocations admitted during preliminary verification. - if err := candidate.VerifyCredential(fenceCtx); err != nil { + if err := m.verifyCredential(fenceCtx, candidate.Setup); err != nil { return deployment.View{}, err } } } - result, err := m.deployment.Update(ctx, m.setupInstallationID, *candidate.Selection) + + result, err := m.deployment.Update(ctx, m.setupInstallationID, candidate.Selection) if err != nil { return deployment.View{}, err } @@ -211,9 +219,7 @@ func (m *runtimeManager) publishEmptyDeployment(installationID string, generatio defer m.mu.Unlock() m.config = RuntimeProvider{InstallationID: installationID, Generation: generation} m.nodes = make(map[string]*runtimeNode) - if m.publishUnconfigured != nil { - m.publishUnconfigured(generation) - } + m.publishSelection(generation, nil) m.switching = false m.switchDrained = nil } diff --git a/services/core/internal/execution/sandbox_deployment_switch_test.go b/services/core/internal/execution/sandbox_deployment_switch_test.go index aa3fb0930..14129ca6b 100644 --- a/services/core/internal/execution/sandbox_deployment_switch_test.go +++ b/services/core/internal/execution/sandbox_deployment_switch_test.go @@ -7,9 +7,7 @@ import ( "testing" "time" - "github.com/MiniMax-AI/OpenAgentCore/internal/sandboxlink/relay" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/deployment" - "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/runtimegateway" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox/docker" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox/node" "github.com/google/uuid" @@ -20,11 +18,12 @@ func TestSandboxManagerSwitchDrainsBeforeDirectActivation(t *testing.T) { defer hub.Close() id := uuid.NewString() config := &RuntimeProvider{InstallationID: id, ProviderKind: "docker", Mode: "nodes", Generation: 1, SandboxLink: "wss://core.example/api/v1/sandbox-link", BackendFingerprint: strings.Repeat("a", 64), Provider: hub.Proxy(uuid.NewString(), docker.Operations(), 1)} - m, err := newRuntimeManager(Owner{Lease: heldLease{}}, nil, nil, nil, runtimegateway.NewRegistry(), relay.New(nil), nil, NewDeferredRuntimeProvider(id, func(context.Context) (*RuntimeProvider, error) { return config, nil }, unusedPreparation(t))) + m, err := testManager(t, Owner{Lease: heldLease{}}, nil, nil, id, config.Provider) if err != nil { t.Fatal(err) } defer func() { m.stop(); m.drain() }() + selectTestProvider(t, m, config) if _, err := m.ensureDeployment(t.Context()); err != nil { t.Fatal(err) } @@ -64,6 +63,7 @@ func TestSandboxManagerSwitchDrainsBeforeDirectActivation(t *testing.T) { t.Fatal("switch drain blocked") } config = &RuntimeProvider{InstallationID: id, ProviderKind: "e2b", Mode: "direct", Generation: 2, SandboxLink: "wss://core.example/api/v1/sandbox-link", BackendFingerprint: strings.Repeat("b", 64), Provider: hub.Proxy(uuid.NewString(), docker.Operations(), 1)} + selectTestProvider(t, m, config) if err := m.activateDeployment(t.Context(), deployment.View{InstallationID: id, Generation: 2, Mode: "direct", Provider: "e2b"}); err != nil { t.Fatal(err) } @@ -81,7 +81,7 @@ func TestSandboxManagerSwitchDrainsBeforeDirectActivation(t *testing.T) { func TestSandboxManagerFailedActivationStaysPaused(t *testing.T) { id := uuid.NewString() - m, err := newRuntimeManager(Owner{Lease: heldLease{}}, nil, nil, nil, runtimegateway.NewRegistry(), relay.New(nil), nil, NewDeferredRuntimeProvider(id, func(context.Context) (*RuntimeProvider, error) { return nil, errors.New("provider unavailable") }, unusedPreparation(t))) + m, err := testManager(t, Owner{Lease: heldLease{}}, nil, nil, id, nil) if err != nil { t.Fatal(err) } @@ -89,6 +89,9 @@ func TestSandboxManagerFailedActivationStaysPaused(t *testing.T) { if err := m.pauseDeployment(t.Context()); err != nil { t.Fatal(err) } + m.setups = &fakeDeploymentSetups{t: t, setup: func(context.Context) (deployment.Setup, error) { + return deployment.Setup{}, errors.New("provider unavailable") + }} if err := m.activateDeployment(t.Context(), deployment.View{InstallationID: id, Generation: 2, Mode: "direct", Provider: "e2b"}); err == nil { t.Fatal("failed provider activated") } @@ -102,10 +105,11 @@ func TestSandboxManagerCancelledSwitchCannotResumeBeforeDrain(t *testing.T) { defer hub.Close() id := uuid.NewString() config := &RuntimeProvider{InstallationID: id, ProviderKind: "docker", Mode: "nodes", Generation: 1, SandboxLink: "wss://core.example/api/v1/sandbox-link", BackendFingerprint: strings.Repeat("a", 64), Provider: hub.Proxy(uuid.NewString(), docker.Operations(), 1)} - m, err := newRuntimeManager(Owner{Lease: heldLease{}}, nil, nil, nil, runtimegateway.NewRegistry(), relay.New(nil), nil, NewDeferredRuntimeProvider(id, func(context.Context) (*RuntimeProvider, error) { return config, nil }, unusedPreparation(t))) + m, err := testManager(t, Owner{Lease: heldLease{}}, nil, nil, id, config.Provider) if err != nil { t.Fatal(err) } + selectTestProvider(t, m, config) if _, err := m.ensureDeployment(t.Context()); err != nil { t.Fatal(err) } @@ -147,6 +151,7 @@ func TestSandboxManagerCancelledSwitchCannotResumeBeforeDrain(t *testing.T) { } finish() released = true + selectTestProvider(t, m, config) if err := m.activateDeployment(t.Context(), expected); err != nil { t.Fatal("drained generation did not resume", err) } @@ -156,11 +161,7 @@ func TestSandboxActivationCannotBypassOutstandingDrain(t *testing.T) { hub := node.NewHub(node.HubOptions{}) defer hub.Close() id := uuid.NewString() - m, err := newRuntimeManager(Owner{Lease: heldLease{}}, nil, nil, nil, runtimegateway.NewRegistry(), relay.New(nil), nil, NewDeferredRuntimeProvider(id, - func(context.Context) (*RuntimeProvider, error) { return nil, nil }, - func(_ context.Context, setup deployment.Setup) (PreparedRuntimeDeployment, error) { - return PreparedRuntimeDeployment{Config: &RuntimeProvider{InstallationID: setup.InstallationID, ProviderKind: setup.Provider, Mode: setup.Mode, SandboxLink: "wss://core.example/api/v1/sandbox-link", BackendFingerprint: setup.BackendFingerprint, Provider: hub.Proxy(uuid.NewString(), docker.Operations(), 1)}}, nil - })) + m, err := testManager(t, Owner{Lease: heldLease{}}, nil, nil, id, nil) if err != nil { t.Fatal(err) } diff --git a/services/core/internal/execution/sandbox_generations_test.go b/services/core/internal/execution/sandbox_generations_test.go index 9b80618b8..43084bdc1 100644 --- a/services/core/internal/execution/sandbox_generations_test.go +++ b/services/core/internal/execution/sandbox_generations_test.go @@ -3,12 +3,12 @@ package execution import ( "context" "errors" + "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox/providers" "testing" + "time" - "github.com/MiniMax-AI/OpenAgentCore/internal/sandboxlink/relay" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/adminaudit" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/deployment" - "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/runtimegateway" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox/docker" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox/e2b" @@ -31,27 +31,36 @@ func TestE2BReplacementVerifiesTwiceAndNeverPublishesFailedCommit(t *testing.T) hub := node.NewHub(node.HubOptions{}) defer hub.Close() provider := hub.Proxy(uuid.NewString(), docker.Operations(), 1) - verifyCalls, published, fenced, released := 0, 0, 0, 0 + verifyCalls := 0 var rejectAt int var rejection error = sandbox.ErrCredentialOwnership - config := NewDeferredRuntimeProvider(id, func(ctx context.Context) (*RuntimeProvider, error) { - setup, err := deployments.Setup(ctx) - if err != nil { - return nil, err - } - return &RuntimeProvider{InstallationID: id, ProviderKind: "e2b", Mode: "direct", Generation: setup.Generation, SandboxLink: "wss://core.example/api/v1/sandbox-link", BackendFingerprint: setup.BackendFingerprint, Provider: provider}, nil - }, func(ctx context.Context, setup deployment.Setup) (PreparedRuntimeDeployment, error) { - return PreparedRuntimeDeployment{Config: &RuntimeProvider{InstallationID: id, ProviderKind: "e2b", Mode: "direct", SandboxLink: "wss://core.example/api/v1/sandbox-link", BackendFingerprint: setup.BackendFingerprint, Provider: provider}, - VerifyCredential: func(context.Context) error { - verifyCalls++ - if verifyCalls == rejectAt { - return rejection - } + m, err := testManager(t, owner, deployments, reader, id, provider) + m.providers = &fakeProviderRegistry{t: t, lookup: providers.Builtin().Lookup, + build: func(sandbox.DirectConfig) (sandbox.SandboxProvider, error) { return provider, nil }, + discover: func(_ context.Context, c sandbox.DirectConfig) (sandbox.Selection, error) { return c.Selection, nil }, + verify: func(ctx context.Context, c sandbox.DirectConfig, _ []sandbox.Reference) error { + if c.Selection.Configuration.(*e2b.DeploymentConfiguration).APIKey != "old-key" { return nil - }, - FenceCredential: func(context.Context) (func(), error) { fenced++; return func() { released++ }, nil }, Publish: func(*RuntimeProvider) { published++ }}, nil - }) - m, err := newRuntimeManager(owner, deployments, reader, nil, runtimegateway.NewRegistry(), relay.New(nil), nil, config) + } + verifyCalls++ + probe, cancel := context.WithTimeout(ctx, time.Millisecond) + defer cancel() + release, err := m.providerCalls.Enter(probe) + if verifyCalls == 1 && err != nil { + t.Error("preliminary verification unexpectedly fenced", err) + } + if verifyCalls == 2 && !errors.Is(err, context.DeadlineExceeded) { + t.Error("final verification was not fenced", err) + } + if release != nil { + release() + } + if verifyCalls == rejectAt { + return rejection + } + return nil + }, + } if err != nil { t.Fatal(err) } @@ -90,9 +99,16 @@ func TestE2BReplacementVerifiesTwiceAndNeverPublishesFailedCommit(t *testing.T) t.Fatal("failure published", failure) } committed, err := deployments.Setup(t.Context()) - if err != nil || committed.Generation != 1 || committed.Configuration.(*e2b.DeploymentConfiguration).APIKey != "old-key" || published != 0 || fenced != released { + if err != nil || committed.Generation != 1 || committed.Configuration.(*e2b.DeploymentConfiguration).APIKey != "old-key" || m.selected.Load().Generation != 1 { t.Fatal("partial credential publication", failure, err) } + probe, cancel := context.WithTimeout(t.Context(), time.Second) + release, fenceErr := m.providerCalls.Enter(probe) + cancel() + if fenceErr != nil { + t.Fatal("failed update retained fence", fenceErr) + } + release() current, err := m.node("") if err != nil || current != old || m.switching { t.Fatal("online failure drained an owned lifecycle", err) @@ -101,8 +117,8 @@ func TestE2BReplacementVerifiesTwiceAndNeverPublishesFailedCommit(t *testing.T) verifyCalls = 0 rejectAt = 0 result, err := worker.UpdateSandboxDeployment(audit, request) - if err != nil || result.Generation != 2 || verifyCalls != 2 || published != 1 || fenced != released { - t.Fatal(result, verifyCalls, published, err) + if err != nil || result.Generation != 2 || verifyCalls != 2 || m.selected.Load().Generation != 2 { + t.Fatal(result, verifyCalls, m.selected.Load(), err) } current, err := m.node("") if err != nil || current != old { diff --git a/services/core/internal/execution/sandbox_provider_contract_test.go b/services/core/internal/execution/sandbox_provider_contract_test.go index c2c4dfe8b..2f54567cd 100644 --- a/services/core/internal/execution/sandbox_provider_contract_test.go +++ b/services/core/internal/execution/sandbox_provider_contract_test.go @@ -2,11 +2,10 @@ package execution import ( "context" + "reflect" "strings" "testing" - "github.com/MiniMax-AI/OpenAgentCore/internal/sandboxlink/relay" - "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/runtimegateway" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox" "github.com/google/uuid" ) @@ -19,16 +18,17 @@ func TestSandboxProviderRegistrationDoesNotRequireAnExecutionVendorBranch(t *tes id := uuid.NewString() config := &RuntimeProvider{InstallationID: id, ProviderKind: "contract-fixture", Mode: mode, SandboxLink: "wss://core.example/api/v1/sandbox-link", BackendFingerprint: strings.Repeat("a", 64), Provider: &lifecycleOnlySandbox{}} - m, err := newRuntimeManager(Owner{Lease: heldLease{}}, nil, nil, nil, runtimegateway.NewRegistry(), relay.New(nil), nil, NewDeferredRuntimeProvider(id, func(context.Context) (*RuntimeProvider, error) { return config, nil }, unusedPreparation(t))) + m, err := testManager(t, Owner{Lease: heldLease{}}, nil, nil, id, config.Provider) if err != nil { t.Fatal(err) } t.Cleanup(func() { m.stop(); m.drain() }) + selectTestProvider(t, m, config) ready, err := m.ensureDeployment(t.Context()) if err != nil || !ready { t.Fatalf("registered provider cannot enter common lifecycle: %v", err) } - if m.config.Provider != config.Provider || m.config.ProviderKind != config.ProviderKind || m.config.Mode != mode { + if m.config.ProviderKind != config.ProviderKind || m.config.Mode != mode || !reflect.DeepEqual(m.config.Provider.ProviderOperations(), config.Provider.ProviderOperations()) { t.Fatal("registration identity changed") } }) diff --git a/services/core/internal/execution/sandbox_reset_test.go b/services/core/internal/execution/sandbox_reset_test.go index c2113fea1..780a81641 100644 --- a/services/core/internal/execution/sandbox_reset_test.go +++ b/services/core/internal/execution/sandbox_reset_test.go @@ -7,7 +7,6 @@ import ( "testing" "time" - "github.com/MiniMax-AI/OpenAgentCore/internal/sandboxlink/relay" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/adminaudit" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/credentialcrypto" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/deployment" @@ -15,7 +14,6 @@ import ( "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/persistence/postgres/pgtest" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/persistence/postgres/pgunit" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/persistence/postgres/sessionpg" - "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/runtimegateway" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox/docker" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox/e2b" @@ -80,18 +78,14 @@ func TestSandboxResetPageTimeoutRecoversCommittedOwner(t *testing.T) { hub := node.NewHub(node.HubOptions{}) defer hub.Close() loads := 0 - config := NewDeferredRuntimeProvider(id, func(ctx context.Context) (*RuntimeProvider, error) { + m, err := testManager(t, owner, deployments, reader, id, hub.Proxy(uuid.NewString(), docker.Operations(), 1)) + m.setups = &fakeDeploymentSetups{t: t, setup: func(ctx context.Context) (deployment.Setup, error) { if ctx.Err() != nil { t.Error("recovery inherited cancelled page") } loads++ - setup, err := deployments.Setup(ctx) - if err != nil || setup.Provider == "" { - return nil, err - } - return &RuntimeProvider{InstallationID: id, ProviderKind: setup.Provider, Mode: setup.Mode, Generation: setup.Generation, SandboxLink: "wss://core.example/api/v1/sandbox-link", BackendFingerprint: setup.BackendFingerprint, Provider: hub.Proxy(uuid.NewString(), docker.Operations(), 1)}, nil - }, unusedPreparation(t)) - m, err := newRuntimeManager(owner, deployments, reader, nil, runtimegateway.NewRegistry(), relay.New(nil), nil, config) + return deployments.Setup(ctx) + }} if err != nil { t.Fatal(err) } @@ -198,21 +192,8 @@ func TestSandboxResetPublishesCommittedGenerationWithoutReading(t *testing.T) { deployments, _ := deploymentOperations(t, &strictDeploymentStorage{t: t}, reader, &strictExecutionStorage{t: t}) hub := node.NewHub(node.HubOptions{}) defer hub.Close() - config := NewDeferredRuntimeProvider(id, func(ctx context.Context) (*RuntimeProvider, error) { - setup, err := pooled.Setup(ctx) - if err != nil || setup.Provider == "" { - return nil, err - } - return &RuntimeProvider{InstallationID: id, ProviderKind: setup.Provider, Mode: setup.Mode, Generation: setup.Generation, SandboxLink: "wss://core.example/api/v1/sandbox-link", BackendFingerprint: setup.BackendFingerprint, Provider: hub.Proxy(uuid.NewString(), docker.Operations(), 1)}, nil - }, unusedPreparation(t)) - var published []uint64 - config.PublishUnconfigured = func(generation uint64) { - if committedReads != 0 { - t.Error("a deployment read stood between the reset commit and its publication") - } - published = append(published, generation) - } - m, err := newRuntimeManager(owner, deployments, adapter, nil, runtimegateway.NewRegistry(), relay.New(nil), nil, config) + m, err := testManager(t, owner, deployments, adapter, id, hub.Proxy(uuid.NewString(), docker.Operations(), 1)) + m.setups = pooled if err != nil { t.Fatal(err) } @@ -227,8 +208,8 @@ func TestSandboxResetPublishesCommittedGenerationWithoutReading(t *testing.T) { if err := m.resetStep(t.Context()); err != nil { t.Fatal(err) } - if len(published) != 1 || published[0] != 2 { - t.Fatal("reset did not publish its committed generation", published) + if published := m.selected.Load(); published == nil || published.Generation != 2 || published.Config != nil || committedReads != 0 { + t.Fatal("reset did not publish its committed generation without reading", published, committedReads) } m.mu.Lock() current := m.config @@ -247,7 +228,7 @@ func TestCommittedResetViewStopsOwnerWithoutLease(t *testing.T) { return deployment.Snapshot{Record: deployment.Record{InstallationID: id, Generation: 1}}, nil }} deployments, operations := deploymentOperations(t, &strictDeploymentStorage{t: t}, reader, &strictExecutionStorage{t: t}) - m, err := newRuntimeManager(Owner{Lease: lostLease{}, Deployment: operations}, deployments, reader, nil, runtimegateway.NewRegistry(), relay.New(nil), nil, NewDeferredRuntimeProvider(id, func(context.Context) (*RuntimeProvider, error) { return nil, nil }, unusedPreparation(t))) + m, err := testManager(t, Owner{Lease: lostLease{}, Deployment: operations}, deployments, reader, id, nil) if err != nil { t.Fatal(err) } @@ -269,7 +250,7 @@ func TestCommittedResetViewStopsOwnerWithoutLease(t *testing.T) { func TestSandboxResetChangesReturnViewReadAfterCommit(t *testing.T) { owner, deployments, reader := resetManager(t) id := initializeE2BDeployment(t, owner) - m, err := newRuntimeManager(owner, deployments, reader, nil, runtimegateway.NewRegistry(), relay.New(nil), nil, NewDeferredRuntimeProvider(id, func(context.Context) (*RuntimeProvider, error) { return nil, nil }, unusedPreparation(t))) + m, err := testManager(t, owner, deployments, reader, id, nil) if err != nil { t.Fatal(err) } diff --git a/services/core/internal/execution/sandbox_snapshot_budget_test.go b/services/core/internal/execution/sandbox_snapshot_budget_test.go index b0f0d71f3..a7b62f9c1 100644 --- a/services/core/internal/execution/sandbox_snapshot_budget_test.go +++ b/services/core/internal/execution/sandbox_snapshot_budget_test.go @@ -8,10 +8,8 @@ import ( "testing" "time" - "github.com/MiniMax-AI/OpenAgentCore/internal/sandboxlink/relay" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/adminaudit" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/deployment" - "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/runtimegateway" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox/docker" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox/node" "github.com/google/uuid" @@ -68,14 +66,7 @@ func TestSandboxResetSnapshotFitsPageBudget(t *testing.T) { id := initializeE2BDeployment(t, owner) hub := node.NewHub(node.HubOptions{}) defer hub.Close() - configuration := NewDeferredRuntimeProvider(id, func(ctx context.Context) (*RuntimeProvider, error) { - setup, err := deployments.Setup(ctx) - if err != nil || setup.Provider == "" { - return nil, err - } - return &RuntimeProvider{InstallationID: id, ProviderKind: setup.Provider, Generation: setup.Generation, Mode: setup.Mode, SandboxLink: "wss://core.example/api/v1/sandbox-link", BackendFingerprint: setup.BackendFingerprint, Provider: hub.Proxy(uuid.NewString(), docker.Operations(), 1)}, nil - }, unusedPreparation(t)) - m, err := newRuntimeManager(owner, deployments, reader, nil, runtimegateway.NewRegistry(), relay.New(nil), nil, configuration) + m, err := testManager(t, owner, deployments, reader, id, hub.Proxy(uuid.NewString(), docker.Operations(), 1)) if err != nil { t.Fatal(err) } diff --git a/services/core/internal/execution/worker.go b/services/core/internal/execution/worker.go index efa7d09c6..5b0cbfe37 100644 --- a/services/core/internal/execution/worker.go +++ b/services/core/internal/execution/worker.go @@ -66,7 +66,7 @@ func StartWorker(ctx context.Context, dispatcher *Dispatcher, owner Owner) (_ *W if dispatcher.SessionsReader == nil { return nil, errors.New("execution worker requires the Session reader") } - if dispatcher.ManagedRuntimes == nil { + if dispatcher.Providers == nil { return nil, errors.New("execution worker requires the sandbox runtime provider") } owned, err := dispatcher.Bind(owner) @@ -81,7 +81,7 @@ func StartWorker(ctx context.Context, dispatcher *Dispatcher, owner Owner) (_ *W } owned.notifications = &executionNotifications{} worker := &Worker{concurrency: dispatcher.MaxConcurrentExecutions, dispatcher: owned, lease: owner.Lease, directoryReads: make(chan directoryReadRequest), fileWrites: make(chan fileWriteRequest), stopped: make(chan struct{}), scheduleWake: make(chan struct{}, 1), connections: &environmentConnections{current: make(map[string]*runtimeConnection), served: make(map[sandboxbootstrap.Resource]uint64)}} - worker.runtimes, err = newRuntimeManager(owner, owned.Deployment, owned.DeploymentReader, owned.SessionsReader, owned.Registry, owned.Links, worker.connections, owned.ManagedRuntimes) + worker.runtimes, err = newRuntimeManager(owner, owned, worker.connections) if err != nil { return nil, err } diff --git a/services/core/internal/sandbox/generation_router.go b/services/core/internal/sandbox/generation_router.go new file mode 100644 index 000000000..f6955f747 --- /dev/null +++ b/services/core/internal/sandbox/generation_router.go @@ -0,0 +1,136 @@ +package sandbox + +import ( + "context" + "maps" + + "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/providercontract" + "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/runtimeobs" +) + +// NewGenerationRouter resolves each allocation's immutable generation for every +// operation. The resolver owns any call fence and releases it after the call. +func NewGenerationRouter(declared providercontract.Operations, resolve func(context.Context, Reference) (SandboxProvider, func(), error)) SandboxProvider { + return &generationRouter{operations: maps.Clone(declared), resolve: resolve} +} + +type generationRouter struct { + operations providercontract.Operations + resolve func(context.Context, Reference) (SandboxProvider, func(), error) +} + +func (p *generationRouter) route(ctx context.Context, operation string, ref Reference) (SandboxProvider, func(), error) { + if err := p.operations[operation].Check(operation); err != nil { + return nil, nil, err + } + return p.resolve(ctx, ref) +} +func (p *generationRouter) Create(ctx context.Context, b Bootstrap) (Info, error) { + v, done, err := p.route(ctx, "Create", b.Reference) + if err != nil { + return Info{}, err + } + defer done() + return v.Create(ctx, b) +} +func (p *generationRouter) GetInfo(ctx context.Context, r Reference) (Info, error) { + v, done, err := p.route(ctx, "GetInfo", r) + if err != nil { + return Info{}, err + } + defer done() + return v.GetInfo(ctx, r) +} +func (p *generationRouter) Renew(ctx context.Context, r Reference) (Info, error) { + v, done, err := p.route(ctx, "Renew", r) + if err != nil { + return Info{}, err + } + defer done() + return v.Renew(ctx, r) +} +func (p *generationRouter) Kill(ctx context.Context, r Reference) error { + v, done, err := p.route(ctx, "Kill", r) + if err != nil { + return err + } + defer done() + return v.Kill(ctx, r) +} + +func (p *generationRouter) ProviderOperations() providercontract.Operations { + return maps.Clone(p.operations) +} +func (p *generationRouter) Observe(ctx context.Context, t runtimeobs.Target) (runtimeobs.Sample, error) { + v, done, err := p.route(ctx, "Observe", Reference{TenantID: t.TenantID, EnvironmentID: t.EnvironmentID, AllocationID: t.Instance.AllocationID}) + if err != nil { + return runtimeobs.Sample{}, err + } + defer done() + return v.Observe(ctx, t) +} + +func (p *generationRouter) Initial(ctx context.Context, r Reference) (Compute, error) { + v, done, err := p.route(ctx, "Initial", r) + if err != nil { + return Compute{}, err + } + defer done() + return v.Initial(ctx, r) +} +func (p *generationRouter) NewCompute(ctx context.Context, r Reference, g uint64, snapshot *SnapshotIdentity) (Compute, error) { + v, done, err := p.route(ctx, "NewCompute", r) + if err != nil { + return Compute{}, err + } + defer done() + return v.NewCompute(ctx, r, g, snapshot) +} +func (p *generationRouter) GetCompute(ctx context.Context, r Reference, c Compute) (ComputeState, error) { + v, done, err := p.route(ctx, "GetCompute", r) + if err != nil { + return ComputeState{}, err + } + defer done() + return v.GetCompute(ctx, r, c) +} +func (p *generationRouter) Suspend(ctx context.Context, q SuspendRequest) (ComputeState, error) { + v, done, err := p.route(ctx, "Suspend", q.Reference) + if err != nil { + return ComputeState{}, err + } + defer done() + return v.Suspend(ctx, q) +} +func (p *generationRouter) Resume(ctx context.Context, q ResumeRequest) (ComputeState, error) { + v, done, err := p.route(ctx, "Resume", q.Reference) + if err != nil { + return ComputeState{}, err + } + defer done() + return v.Resume(ctx, q) +} +func (p *generationRouter) KillCompute(ctx context.Context, r Reference, c Compute) error { + v, done, err := p.route(ctx, "KillCompute", r) + if err != nil { + return err + } + defer done() + return v.KillCompute(ctx, r, c) +} +func (p *generationRouter) DeleteSnapshot(ctx context.Context, r Reference, snapshot SnapshotIdentity) error { + v, done, err := p.route(ctx, "DeleteSnapshot", r) + if err != nil { + return err + } + defer done() + return v.DeleteSnapshot(ctx, r, snapshot) +} +func (p *generationRouter) ResumeCompute(ctx context.Context, r Reference, c Compute) (ComputeState, error) { + v, done, err := p.route(ctx, "ResumeCompute", r) + if err != nil { + return ComputeState{}, err + } + defer done() + return v.ResumeCompute(ctx, r, c) +} diff --git a/services/core/internal/sandbox/generation_router_test.go b/services/core/internal/sandbox/generation_router_test.go new file mode 100644 index 000000000..37a08e0eb --- /dev/null +++ b/services/core/internal/sandbox/generation_router_test.go @@ -0,0 +1,152 @@ +package sandbox_test + +import ( + "context" + "errors" + "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/providercontract" + "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/runtimeobs" + "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox" + "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox/microsandbox" + "reflect" + "testing" +) + +type routedProvider struct { + t *testing.T + operation string + args []any + calls int + released *bool + failure error +} + +func (p *routedProvider) called(operation string, args ...any) error { + p.t.Helper() + if operation != p.operation || !reflect.DeepEqual(args, p.args) || *p.released { + p.t.Fatalf("routed call changed: %s %v", operation, args) + } + p.calls++ + return p.failure +} +func (p *routedProvider) ProviderOperations() providercontract.Operations { + return microsandbox.Operations() +} +func (p *routedProvider) Create(_ context.Context, b sandbox.Bootstrap) (sandbox.Info, error) { + return sandbox.Info{}, p.called("Create", b) +} +func (p *routedProvider) GetInfo(_ context.Context, r sandbox.Reference) (sandbox.Info, error) { + return sandbox.Info{}, p.called("GetInfo", r) +} +func (p *routedProvider) Renew(_ context.Context, r sandbox.Reference) (sandbox.Info, error) { + return sandbox.Info{}, p.called("Renew", r) +} +func (p *routedProvider) Kill(_ context.Context, r sandbox.Reference) error { + return p.called("Kill", r) +} +func (p *routedProvider) Observe(_ context.Context, t runtimeobs.Target) (runtimeobs.Sample, error) { + return runtimeobs.Sample{}, p.called("Observe", t) +} +func (p *routedProvider) Initial(_ context.Context, r sandbox.Reference) (sandbox.Compute, error) { + return sandbox.Compute{}, p.called("Initial", r) +} +func (p *routedProvider) NewCompute(_ context.Context, r sandbox.Reference, g uint64, s *sandbox.SnapshotIdentity) (sandbox.Compute, error) { + return sandbox.Compute{}, p.called("NewCompute", r, g, s) +} +func (p *routedProvider) GetCompute(_ context.Context, r sandbox.Reference, c sandbox.Compute) (sandbox.ComputeState, error) { + return sandbox.ComputeState{}, p.called("GetCompute", r, c) +} +func (p *routedProvider) Suspend(_ context.Context, q sandbox.SuspendRequest) (sandbox.ComputeState, error) { + return sandbox.ComputeState{}, p.called("Suspend", q) +} +func (p *routedProvider) Resume(_ context.Context, q sandbox.ResumeRequest) (sandbox.ComputeState, error) { + return sandbox.ComputeState{}, p.called("Resume", q) +} +func (p *routedProvider) KillCompute(_ context.Context, r sandbox.Reference, c sandbox.Compute) error { + return p.called("KillCompute", r, c) +} +func (p *routedProvider) DeleteSnapshot(_ context.Context, r sandbox.Reference, s sandbox.SnapshotIdentity) error { + return p.called("DeleteSnapshot", r, s) +} +func (p *routedProvider) ResumeCompute(_ context.Context, r sandbox.Reference, c sandbox.Compute) (sandbox.ComputeState, error) { + return sandbox.ComputeState{}, p.called("ResumeCompute", r, c) +} + +func TestGenerationRouterRoutesEveryOperationAndReleasesCalls(t *testing.T) { + ref := sandbox.Reference{TenantID: "tenant", EnvironmentID: "environment", AllocationID: "allocation"} + compute := sandbox.Compute{ID: "compute", Generation: 7} + snapshot := sandbox.SnapshotIdentity{} + target := runtimeobs.Target{TenantID: ref.TenantID, EnvironmentID: ref.EnvironmentID, Instance: runtimeobs.Instance{AllocationID: ref.AllocationID}} + for _, test := range []struct { + name string + args []any + call func(sandbox.SandboxProvider) error + }{ + {"Create", []any{sandbox.Bootstrap{Reference: ref}}, func(p sandbox.SandboxProvider) error { + _, err := p.Create(t.Context(), sandbox.Bootstrap{Reference: ref}) + return err + }}, + {"GetInfo", []any{ref}, func(p sandbox.SandboxProvider) error { _, err := p.GetInfo(t.Context(), ref); return err }}, + {"Renew", []any{ref}, func(p sandbox.SandboxProvider) error { _, err := p.Renew(t.Context(), ref); return err }}, + {"Kill", []any{ref}, func(p sandbox.SandboxProvider) error { return p.Kill(t.Context(), ref) }}, + {"Observe", []any{target}, func(p sandbox.SandboxProvider) error { _, err := p.Observe(t.Context(), target); return err }}, + {"Initial", []any{ref}, func(p sandbox.SandboxProvider) error { _, err := p.Initial(t.Context(), ref); return err }}, + {"NewCompute", []any{ref, uint64(9), &snapshot}, func(p sandbox.SandboxProvider) error { + _, err := p.NewCompute(t.Context(), ref, uint64(9), &snapshot) + return err + }}, + {"GetCompute", []any{ref, compute}, func(p sandbox.SandboxProvider) error { _, err := p.GetCompute(t.Context(), ref, compute); return err }}, + {"Suspend", []any{sandbox.SuspendRequest{Reference: ref}}, func(p sandbox.SandboxProvider) error { + _, err := p.Suspend(t.Context(), sandbox.SuspendRequest{Reference: ref}) + return err + }}, + {"Resume", []any{sandbox.ResumeRequest{Reference: ref}}, func(p sandbox.SandboxProvider) error { + _, err := p.Resume(t.Context(), sandbox.ResumeRequest{Reference: ref}) + return err + }}, + {"KillCompute", []any{ref, compute}, func(p sandbox.SandboxProvider) error { return p.KillCompute(t.Context(), ref, compute) }}, + {"DeleteSnapshot", []any{ref, snapshot}, func(p sandbox.SandboxProvider) error { return p.DeleteSnapshot(t.Context(), ref, snapshot) }}, + {"ResumeCompute", []any{ref, compute}, func(p sandbox.SandboxProvider) error { + _, err := p.ResumeCompute(t.Context(), ref, compute) + return err + }}, + } { + t.Run(test.name, func(t *testing.T) { + for _, failure := range []error{nil, errors.New("native failure")} { + released := false + native := &routedProvider{t: t, operation: test.name, args: test.args, released: &released, failure: failure} + resolutions := 0 + operations := microsandbox.Operations() + router := sandbox.NewGenerationRouter(operations, func(_ context.Context, got sandbox.Reference) (sandbox.SandboxProvider, func(), error) { + if got != ref { + t.Fatal("wrong allocation", got) + } + resolutions++ + return native, func() { + if released { + t.Error("double release") + } + released = true + }, nil + }) + operations[test.name] = providercontract.Support{State: providercontract.Unsupported, Reason: "test_rejection"} + if err := test.call(router); !errors.Is(err, failure) || resolutions != 1 || native.calls != 1 || !released { + t.Fatal("lost outcome or release", err, resolutions, native.calls, released) + } + router = sandbox.NewGenerationRouter(operations, func(context.Context, sandbox.Reference) (sandbox.SandboxProvider, func(), error) { + t.Fatal("unsupported operation resolved") + return nil, nil, nil + }) + if err := test.call(router); !errors.Is(err, providercontract.ErrUnsupported) { + t.Fatal(err) + } + resolutionFailure := errors.New("allocation lookup failed") + router = sandbox.NewGenerationRouter(microsandbox.Operations(), func(context.Context, sandbox.Reference) (sandbox.SandboxProvider, func(), error) { + return nil, nil, resolutionFailure + }) + if err := test.call(router); !errors.Is(err, resolutionFailure) { + t.Fatal("lost resolution error", err) + } + } + }) + } +} diff --git a/services/core/internal/sandbox/node/generation_connection_test.go b/services/core/internal/sandbox/node/generation_connection_test.go index 1c3e59d6c..c1356a3b0 100644 --- a/services/core/internal/sandbox/node/generation_connection_test.go +++ b/services/core/internal/sandbox/node/generation_connection_test.go @@ -63,7 +63,7 @@ func TestGenerationWireRoutesOldOwnershipAndCurrentTargetSeparately(t *testing.T }() wait(t, func() bool { return hub.Online(id.NodeID) }) for _, generation := range []uint64{1, 17, 9} { - proxy := hub.GenerationProvider(docker.Operations(), func(context.Context, sandbox.Reference) (string, uint64, error) { return id.NodeID, generation, nil }) + proxy := hub.Proxy(id.NodeID, docker.Operations(), generation) ref := reference() if _, err := proxy.GetInfo(ctx, ref); err != nil { t.Fatal("retained generation info failed", generation, err) diff --git a/services/core/internal/sandbox/node/observations.go b/services/core/internal/sandbox/node/observations.go index 7129d2f33..035869f46 100644 --- a/services/core/internal/sandbox/node/observations.go +++ b/services/core/internal/sandbox/node/observations.go @@ -17,7 +17,7 @@ func validObservation(target runtimeobs.Target, reference sandbox.Reference) boo validID(target.Instance.ProviderKey) && observationReference(target) == reference && target.TokenUsage == nil } -// Observe uses the allocation's existing node resolver and never changes its +// Observe reads the bound node and generation and never changes their // compute state. Session token counters stay in Core, outside provider telemetry. func (p *provider) Observe(ctx context.Context, target runtimeobs.Target) (runtimeobs.Sample, error) { target.TokenUsage = nil diff --git a/services/core/internal/sandbox/node/observations_test.go b/services/core/internal/sandbox/node/observations_test.go index 23a5cbfdb..c5ca75325 100644 --- a/services/core/internal/sandbox/node/observations_test.go +++ b/services/core/internal/sandbox/node/observations_test.go @@ -89,11 +89,11 @@ func TestObservationsRouteThroughAssignedNodeWithoutLifecycleCalls(t *testing.T) stopSecond := runObservationNode(t, hub, server.URL, second, b) ra, rb := reference(), reference() assignments := map[sandbox.Reference]string{ra: first.NodeID, rb: second.NodeID} - source := hub.GenerationProvider(docker.Operations(), func(_ context.Context, r sandbox.Reference) (string, uint64, error) { + source := sandbox.NewGenerationRouter(docker.Operations(), func(_ context.Context, r sandbox.Reference) (sandbox.SandboxProvider, func(), error) { if id, ok := assignments[r]; ok { - return id, 1, nil + return hub.Proxy(id, docker.Operations(), 1), func() {}, nil } - return "", 0, sandbox.ErrOwnership + return nil, nil, sandbox.ErrOwnership }) for _, test := range []struct { ref sandbox.Reference diff --git a/services/core/internal/sandbox/node/operations_test.go b/services/core/internal/sandbox/node/operations_test.go index c273e64e9..925d5a6cc 100644 --- a/services/core/internal/sandbox/node/operations_test.go +++ b/services/core/internal/sandbox/node/operations_test.go @@ -37,10 +37,10 @@ func TestUnsupportedWireIsExplicitAndDoesNotInvokeProvider(t *testing.T) { } } func TestUnsupportedProxyRejectsBeforeNodeResolution(t *testing.T) { - p := (&Hub{}).GenerationProvider(docker.Operations(), func(context.Context, sandbox.Reference) (string, uint64, error) { + p := sandbox.NewGenerationRouter(docker.Operations(), func(context.Context, sandbox.Reference) (sandbox.SandboxProvider, func(), error) { t.Fatal("unsupported call resolved a node") - return "", 0, nil - }).(*provider) + return nil, nil, nil + }) if err := sandbox.ValidateProvider(p); err != nil { t.Fatal(err) } diff --git a/services/core/internal/sandbox/node/proxy.go b/services/core/internal/sandbox/node/proxy.go index a0f6b29cf..647558556 100644 --- a/services/core/internal/sandbox/node/proxy.go +++ b/services/core/internal/sandbox/node/proxy.go @@ -10,30 +10,27 @@ import ( ) type provider struct { - hub *Hub - resolveGeneration func(context.Context, sandbox.Reference) (string, uint64, error) - operations providercontract.Operations + hub *Hub + nodeID string + generation uint64 + operations providercontract.Operations } var _ sandbox.SandboxProvider = (*provider)(nil) // Proxy binds a fixed node and deployment generation explicitly. func (h *Hub) Proxy(id string, declared providercontract.Operations, generation uint64) sandbox.SandboxProvider { - return h.GenerationProvider(declared, func(context.Context, sandbox.Reference) (string, uint64, error) { return id, generation, nil }) + return &provider{hub: h, operations: maps.Clone(declared), nodeID: id, generation: generation} } func (p *provider) call(ctx context.Context, q request) (response, error) { if err := providercontract.Require(p, operationMethod(q.Operation)); err != nil { return response{}, err } - id, generation, err := p.resolveGeneration(ctx, q.Reference) - if err != nil { - return response{}, err - } - if !validID(id) || !validGeneration(generation) { + if !validID(p.nodeID) || !validGeneration(p.generation) { return response{}, sandbox.ErrOwnership } - q.DeploymentGeneration = generation - out, err := p.hub.call(ctx, id, q) + q.DeploymentGeneration = p.generation + out, err := p.hub.call(ctx, p.nodeID, q) if errors.Is(err, providercontract.ErrUnsupported) { if _, valid := providercontract.UnsupportedReason(err, operationMethod(q.Operation)); !valid { return response{}, sandbox.ErrComputeUnconfirmed @@ -132,10 +129,3 @@ func (p *provider) DeleteSnapshot(ctx context.Context, r sandbox.Reference, s sa func (p *provider) ResumeCompute(ctx context.Context, r sandbox.Reference, c sandbox.Compute) (sandbox.ComputeState, error) { return p.state(ctx, request{Operation: "resume_compute", Reference: r, Compute: &c}) } - -// GenerationProvider routes every operation with allocation-owned generation, -// distinct from the request's compute generation. declared is the kind's -// registered operations. -func (h *Hub) GenerationProvider(declared providercontract.Operations, resolve func(context.Context, sandbox.Reference) (string, uint64, error)) sandbox.SandboxProvider { - return &provider{hub: h, operations: maps.Clone(declared), resolveGeneration: resolve} -} diff --git a/services/core/tests/integration/admin_session_archive_worker_http_test.go b/services/core/tests/integration/admin_session_archive_worker_http_test.go index 1005a35b5..6b4c1eeed 100644 --- a/services/core/tests/integration/admin_session_archive_worker_http_test.go +++ b/services/core/tests/integration/admin_session_archive_worker_http_test.go @@ -17,7 +17,6 @@ import ( "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/adminaudit" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/api" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/deployment" - "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/execution" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/runtimedevice" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/runtimegateway" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox" @@ -32,19 +31,7 @@ func TestAdminSessionArchiveWorkerHTTPPostgres(t *testing.T) { s, pool := newManagedTestStore(t) installation := uuid.NewString() provider := &lifecycleProvider{resources: map[string]sandbox.Info{}} - deployments := deploymentService(t, s) - providerConfig := func(setup deployment.Setup) *execution.RuntimeProvider { - return &execution.RuntimeProvider{InstallationID: setup.InstallationID, ProviderKind: setup.Provider, Generation: setup.Generation, Mode: setup.Mode, SandboxLink: "wss://core.example/api/v1/sandbox-link", BackendFingerprint: setup.BackendFingerprint, Provider: provider} - } - configuration := execution.NewDeferredRuntimeProvider(installation, func(ctx context.Context) (*execution.RuntimeProvider, error) { - setup, err := deployments.Setup(ctx) - if err != nil || setup.Provider == "" { - return nil, err - } - return providerConfig(setup), nil - }, func(_ context.Context, setup deployment.Setup) (execution.PreparedRuntimeDeployment, error) { - return execution.PreparedRuntimeDeployment{Config: providerConfig(setup)}, nil - }) + lease, err := pgunit.AcquireLease(t.Context(), pool) if err != nil { t.Fatal(err) @@ -53,7 +40,7 @@ func TestAdminSessionArchiveWorkerHTTPPostgres(t *testing.T) { if err != nil { t.Fatal(errors.Join(err, lease.Close(t.Context()))) } - worker := startOwnedWorker(t, t.Context(), s, &execution.Dispatcher{Registry: runtimegateway.NewRegistry(), ManagedRuntimes: configuration}, owner) + worker := startOwnedWorker(t, t.Context(), s, webDispatcher(t, installation, provider, runtimegateway.NewRegistry(), nil), owner) var once sync.Once stop := func() { once.Do(func() { diff --git a/services/core/tests/integration/archive_cancellation_cleanup_test.go b/services/core/tests/integration/archive_cancellation_cleanup_test.go index a7ff9f6ed..9815f9620 100644 --- a/services/core/tests/integration/archive_cancellation_cleanup_test.go +++ b/services/core/tests/integration/archive_cancellation_cleanup_test.go @@ -29,7 +29,7 @@ func TestArchiveCancellationThenDeleteRemovesHome(t *testing.T) { Reference: sandbox.Reference{TenantID: h.tenant, EnvironmentID: h.resource.EnvironmentID, AllocationID: h.resource.ID}, ProviderID: h.resource.ID, State: "running", BootstrapComplete: true, CreateSettled: true, }}} - h.d.ManagedRuntimes = webRuntimes(t, h.s, setup.InstallationID, provider, nil) + h.d = webDispatcher(t, setup.InstallationID, provider, h.d.Registry, h.d.Links) owner := h.owner() worker := startOwnedWorker(t, t.Context(), h.s, h.d, owner) runWorker(t, worker) diff --git a/services/core/tests/integration/credential_matrix_http_test.go b/services/core/tests/integration/credential_matrix_http_test.go index 071fbcb10..cad4049d1 100644 --- a/services/core/tests/integration/credential_matrix_http_test.go +++ b/services/core/tests/integration/credential_matrix_http_test.go @@ -13,7 +13,6 @@ import ( "github.com/MiniMax-AI/OpenAgentCore/internal/sandboxlink/relay" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/api" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/deployment" - "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/execution" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/persistence/postgres/pgunit" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/persistence/postgres/projectpg" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/projects" @@ -92,19 +91,10 @@ func TestCredentialNamespaceMatrix(t *testing.T) { created("POST", "/core/v1/projects/"+project.ID+"/environments/"+environment.ID+"/executor-credentials", coreKey, `{"key_id":"`+uuid.NewString()+`"}`, &executor) // A node credential: a Docker deployment, an enrollment token issued with the Core key, and an enrolled node. - deployments := deploymentService(t, s) installation := uuid.NewString() provider := &lifecycleProvider{resources: map[string]sandbox.Info{}} - runtimes := execution.NewDeferredRuntimeProvider(installation, func(ctx context.Context) (*execution.RuntimeProvider, error) { - setup, err := deployments.Setup(ctx) - if err != nil || setup.Provider == "" { - return nil, err - } - return &execution.RuntimeProvider{InstallationID: setup.InstallationID, ProviderKind: setup.Provider, Mode: setup.Mode, BackendFingerprint: setup.BackendFingerprint, SandboxLink: "wss://core.example/api/v1/sandbox-link", Provider: provider}, nil - }, func(_ context.Context, setup deployment.Setup) (execution.PreparedRuntimeDeployment, error) { - return execution.PreparedRuntimeDeployment{Config: &execution.RuntimeProvider{InstallationID: setup.InstallationID, ProviderKind: setup.Provider, Mode: setup.Mode, SandboxLink: "wss://core.example/api/v1/sandbox-link", BackendFingerprint: setup.BackendFingerprint, Provider: provider}}, nil - }) - worker := startWorker(t, ctx, s, &execution.Dispatcher{Registry: runtimegateway.NewRegistry(), ManagedRuntimes: runtimes}) + + worker := startWorker(t, ctx, s, webDispatcher(t, installation, provider, runtimegateway.NewRegistry(), nil)) var stop sync.Once t.Cleanup(func() { stop.Do(func() { diff --git a/services/core/tests/integration/environment_initialization_test.go b/services/core/tests/integration/environment_initialization_test.go index 8c25f1bd6..ac439bc1f 100644 --- a/services/core/tests/integration/environment_initialization_test.go +++ b/services/core/tests/integration/environment_initialization_test.go @@ -74,7 +74,7 @@ func TestUserManagedPreparationUsesAuthenticatedRuntimeWithoutAllocation(t *test t.Fatal(err) } registry := runtimegateway.NewRegistry() - handler := runtimegateway.NewHandler(runtimegateway.HandlerConfig{Authenticator: runtimegateway.NewAuthenticator(sessionAdapter(s)), Registry: registry}) + handler := runtimegateway.NewHandler(runtimegateway.HandlerConfig{Authenticator: runtimegateway.NewAuthenticator(sessionAdapter(s)), Registry: registry, Links: runtimegateway.NewLinkAuthority(sessionAdapter(s))}) server := httptest.NewServer(http.HandlerFunc(handler.WS)) defer server.Close() link := sandboxlinktest.StartRelay(t, runtimegateway.NewLinkAuthority(sessionAdapter(s))) diff --git a/services/core/tests/integration/link_authority_test.go b/services/core/tests/integration/link_authority_test.go index f8e2c732a..1c485820e 100644 --- a/services/core/tests/integration/link_authority_test.go +++ b/services/core/tests/integration/link_authority_test.go @@ -433,7 +433,7 @@ func TestLinkAuthorityDestroyedAllocation(t *testing.T) { tenant, session, environment := managedSession(t, s) srv := startLinkRoute(t, s) provider := &lifecycleProvider{resources: map[string]sandbox.Info{}} - w := startWorker(t, t.Context(), s, &execution.Dispatcher{Registry: runtimegateway.NewRegistry(), Links: srv.Relay, ManagedRuntimes: webRuntimes(t, s, key, provider, nil)}) + w := startWorker(t, t.Context(), s, webDispatcher(t, key, provider, runtimegateway.NewRegistry(), srv.Relay)) t.Cleanup(func() { ctx, cancel := context.WithCancel(context.Background()); cancel(); _ = w.Run(ctx) }) owner, err := w.ProvisionEnvironment(t.Context(), tenant, environment.ID, key) if err != nil { diff --git a/services/core/tests/integration/runtime_compute_lifecycle_test.go b/services/core/tests/integration/runtime_compute_lifecycle_test.go index 0f33e84a9..950d4cdc8 100644 --- a/services/core/tests/integration/runtime_compute_lifecycle_test.go +++ b/services/core/tests/integration/runtime_compute_lifecycle_test.go @@ -54,7 +54,7 @@ func newFakeCheckpointProvider(t *testing.T, s *Store) *fakeCheckpointProvider { t.Helper() p := &fakeCheckpointProvider{t: t, lifecycleProvider: lifecycleProvider{resources: map[string]sandbox.Info{}}, computes: map[string]sandbox.ComputeState{}, snapshots: map[string]sandbox.SnapshotIdentity{}, bootstraps: map[string]sandbox.Bootstrap{}, serving: map[string]*linkServe{}, host: registerAgentHost(t, s), registry: runtimegateway.NewRegistry(), link: sandboxlinktest.StartRelay(t, runtimegateway.NewLinkAuthority(sessionAdapter(s)))} - handler := runtimegateway.NewHandler(runtimegateway.HandlerConfig{Authenticator: runtimegateway.NewAuthenticator(sessionAdapter(s)), Registry: p.registry}) + handler := runtimegateway.NewHandler(runtimegateway.HandlerConfig{Authenticator: runtimegateway.NewAuthenticator(sessionAdapter(s)), Registry: p.registry, Links: runtimegateway.NewLinkAuthority(sessionAdapter(s))}) server := httptest.NewServer(http.HandlerFunc(handler.WS)) p.endpoint = "ws" + strings.TrimPrefix(server.URL, "http") t.Cleanup(func() { @@ -347,13 +347,12 @@ type computeLifecycleFixture struct { stop func() key string node string - policy execution.RuntimeSuspensionPolicy } func newComputeLifecycleFixture(t *testing.T, maxActive, maxRetained int) *computeLifecycleFixture { t.Helper() s, _ := newManagedTestStore(t) - f := &computeLifecycleFixture{t: t, store: s, provider: newFakeCheckpointProvider(t, s), key: webDeployment(t, s, "microsandbox"), policy: execution.RuntimeSuspensionPolicy{IdleTimeout: time.Second, Retention: time.Hour}} + f := &computeLifecycleFixture{t: t, store: s, provider: newFakeCheckpointProvider(t, s), key: webDeployment(t, s, "microsandbox")} view, err := deploymentService(t, s).View(t.Context()) if err != nil { t.Fatal(err) @@ -365,7 +364,7 @@ func newComputeLifecycleFixture(t *testing.T, maxActive, maxRetained int) *compu func (f *computeLifecycleFixture) start() { t := f.t t.Helper() - w := startWebWorker(t, f.store, f.provider.registry, f.provider.link.Relay, f.key, f.provider, &f.policy) + w := startWebWorker(t, f.store, f.provider.registry, f.provider.link.Relay, f.key, f.provider) // The Worker's claim starts a new owner epoch, in which the node reconnects. onlineManagerNode(t, f.store, f.node) var once sync.Once @@ -418,8 +417,13 @@ func (f *computeLifecycleFixture) phase(tenant, environment, phase string) deplo func (f *computeLifecycleFixture) complete(owner deployment.Allocation) string { id := uuid.NewString() assignSession(f.t, f.store, owner.SessionID, f.provider.host.ID) - f.sql(`INSERT INTO turns(id,session_id,status,completed_at) VALUES($1,$2,'completed',clock_timestamp()-interval '2 minutes')`, id, owner.SessionID) - f.sql(`UPDATE runtime_allocations SET compute_activity_at=clock_timestamp()-interval '2 minutes' WHERE id=$1`, owner.ID) + view, err := deploymentService(f.t, f.store).View(f.t.Context()) + if err != nil { + f.t.Fatal(err) + } + idleSeconds := view.Suspension.IdleSeconds + 60 + f.sql(`INSERT INTO turns(id,session_id,status,completed_at) VALUES($1,$2,'completed',clock_timestamp()-make_interval(secs => $3))`, id, owner.SessionID, idleSeconds) + f.sql(`UPDATE runtime_allocations SET compute_activity_at=clock_timestamp()-make_interval(secs => $2) WHERE id=$1`, owner.ID, idleSeconds) return id } func (f *computeLifecycleFixture) queued(owner deployment.Allocation) string { diff --git a/services/core/tests/integration/runtime_connection_test.go b/services/core/tests/integration/runtime_connection_test.go index 8d3089958..76473c733 100644 --- a/services/core/tests/integration/runtime_connection_test.go +++ b/services/core/tests/integration/runtime_connection_test.go @@ -24,7 +24,7 @@ func TestManagedRuntimeConnectionFollowsServe(t *testing.T) { link := sandboxlinktest.StartRelay(t, runtimegateway.NewLinkAuthority(sessionAdapter(s))) p := &lifecycleProvider{resources: map[string]sandbox.Info{}} start := func() (*execution.Worker, func()) { - w, err := startNextWorker(t, t.Context(), s, &execution.Dispatcher{Registry: runtimegateway.NewRegistry(), Links: link.Relay, ManagedRuntimes: webRuntimes(t, s, key, p, nil)}) + w, err := startNextWorker(t, t.Context(), s, webDispatcher(t, key, p, runtimegateway.NewRegistry(), link.Relay)) if err != nil { t.Fatal(err) } diff --git a/services/core/tests/integration/runtime_deployment_worker_test.go b/services/core/tests/integration/runtime_deployment_worker_test.go index 7cc5c0e85..2e1434b0f 100644 --- a/services/core/tests/integration/runtime_deployment_worker_test.go +++ b/services/core/tests/integration/runtime_deployment_worker_test.go @@ -5,7 +5,6 @@ import ( "testing" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/deployment" - "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/execution" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/runtimegateway" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox" "github.com/google/uuid" @@ -22,7 +21,7 @@ func TestManagedDeploymentStartupRejectsSwitchBeforeBackendAccess(t *testing.T) } stop() replacement := &lifecycleProvider{resources: map[string]sandbox.Info{}} - _, err = startNextWorker(t, t.Context(), s, &execution.Dispatcher{Registry: runtimegateway.NewRegistry(), ManagedRuntimes: webRuntimes(t, s, uuid.NewString(), replacement, nil)}) + _, err = startNextWorker(t, t.Context(), s, webDispatcher(t, uuid.NewString(), replacement, runtimegateway.NewRegistry(), nil)) if !errors.Is(err, deployment.ErrConflict) { t.Fatal("startup switched the claimed installation", err) } diff --git a/services/core/tests/integration/runtime_lifecycle_test.go b/services/core/tests/integration/runtime_lifecycle_test.go index 2e91174f4..cbaf91308 100644 --- a/services/core/tests/integration/runtime_lifecycle_test.go +++ b/services/core/tests/integration/runtime_lifecycle_test.go @@ -78,11 +78,11 @@ func managedWorker(t *testing.T, s *Store, key string, p sandbox.SandboxProvider // managedWorkerMode is managedWorker that also runs the Worker when run is set. func managedWorkerMode(t *testing.T, s *Store, key string, p sandbox.SandboxProvider, run bool) (*execution.Worker, func()) { t.Helper() - dispatcher := &execution.Dispatcher{Registry: runtimegateway.NewRegistry(), ManagedRuntimes: webRuntimes(t, s, key, p, nil)} + dispatcher := webDispatcher(t, key, p, runtimegateway.NewRegistry(), nil) if peer, ok := p.(interface { setRuntimeGateway(*testing.T, *Store, string, *runtimegateway.Registry, *sandboxlinktest.Server) }); ok { - handler := runtimegateway.NewHandler(runtimegateway.HandlerConfig{Authenticator: runtimegateway.NewAuthenticator(sessionAdapter(s)), Registry: dispatcher.Registry}) + handler := runtimegateway.NewHandler(runtimegateway.HandlerConfig{Authenticator: runtimegateway.NewAuthenticator(sessionAdapter(s)), Registry: dispatcher.Registry, Links: runtimegateway.NewLinkAuthority(sessionAdapter(s))}) server := httptest.NewServer(http.HandlerFunc(handler.WS)) t.Cleanup(server.Close) link := sandboxlinktest.StartRelay(t, runtimegateway.NewLinkAuthority(sessionAdapter(s))) diff --git a/services/core/tests/integration/runtime_node_lifecycle_fixture_test.go b/services/core/tests/integration/runtime_node_lifecycle_fixture_test.go index 6f7f07df4..7786362bf 100644 --- a/services/core/tests/integration/runtime_node_lifecycle_fixture_test.go +++ b/services/core/tests/integration/runtime_node_lifecycle_fixture_test.go @@ -92,8 +92,8 @@ func newNodeIsolationFixture(t *testing.T, mode string) *nodeIsolationFixture { }} f := &nodeIsolationFixture{initializationCancel: cancelPreparation, t: t, store: s, nodes: deploymentService(t, s), pool: pool, provider: p, key: webDeployment(t, s, "microsandbox"), nodeA: uuid.NewString(), nodeB: uuid.NewString()} // Keep restored compute awake throughout the isolation assertions. - // The suspension setup explicitly dates its activity two minutes in the past. - f.worker = startWebWorker(t, s, cp.registry, cp.link.Relay, f.key, p, &execution.RuntimeSuspensionPolicy{IdleTimeout: time.Minute, Retention: time.Hour}) + // The suspension setup dates its activity before the committed idle timeout. + f.worker = startWebWorker(t, s, cp.registry, cp.link.Relay, f.key, p) t.Cleanup(f.stop) f.epoch = fixtureOwnerEpoch(t, s) f.enroll(f.nodeA) diff --git a/services/core/tests/integration/runtime_node_lifecycle_test.go b/services/core/tests/integration/runtime_node_lifecycle_test.go index 950e72104..9754ace64 100644 --- a/services/core/tests/integration/runtime_node_lifecycle_test.go +++ b/services/core/tests/integration/runtime_node_lifecycle_test.go @@ -22,10 +22,15 @@ func TestManagedNodesIsolateBlockedProviderAndInitialization(t *testing.T) { wakeOwner := f.provision(wakeTenant, wakeEnv) f.phase(wakeTenant, wakeEnv.ID, "running") assignSession(t, f.store, wakeOwner.SessionID, f.provider.host.ID) - if _, err := f.pool.Exec(t.Context(), "INSERT INTO turns(id,session_id,status,completed_at) VALUES($1,$2,'completed',clock_timestamp()-interval '2 minutes')", uuid.NewString(), wakeOwner.SessionID); err != nil { + view, err := f.nodes.View(t.Context()) + if err != nil { t.Fatal(err) } - if _, err := f.pool.Exec(t.Context(), "UPDATE runtime_allocations SET compute_activity_at=clock_timestamp()-interval '2 minutes' WHERE id=$1", wakeOwner.ID); err != nil { + idleSeconds := view.Suspension.IdleSeconds + 60 + if _, err := f.pool.Exec(t.Context(), "INSERT INTO turns(id,session_id,status,completed_at) VALUES($1,$2,'completed',clock_timestamp()-make_interval(secs => $3))", uuid.NewString(), wakeOwner.SessionID, idleSeconds); err != nil { + t.Fatal(err) + } + if _, err := f.pool.Exec(t.Context(), "UPDATE runtime_allocations SET compute_activity_at=clock_timestamp()-make_interval(secs => $2) WHERE id=$1", wakeOwner.ID, idleSeconds); err != nil { t.Fatal(err) } f.phase(wakeTenant, wakeEnv.ID, "suspended") diff --git a/services/core/tests/integration/runtime_pending_test.go b/services/core/tests/integration/runtime_pending_test.go index 03daea275..6b5a3ddce 100644 --- a/services/core/tests/integration/runtime_pending_test.go +++ b/services/core/tests/integration/runtime_pending_test.go @@ -25,7 +25,7 @@ func TestManagedRuntimeAutomaticBootstrapRecoversCommittedSessions(t *testing.T) t.Fatal(err) } p := &lifecycleProvider{resources: map[string]sandbox.Info{}} - start := func() *execution.Worker { return startWebWorker(t, s, runtimegateway.NewRegistry(), nil, key, p, nil) } + start := func() *execution.Worker { return startWebWorker(t, s, runtimegateway.NewRegistry(), nil, key, p) } stop := func(w *execution.Worker) { ctx, cancel := context.WithCancel(context.Background()) cancel() diff --git a/services/core/tests/integration/runtime_wake_hint_integration_test.go b/services/core/tests/integration/runtime_wake_hint_integration_test.go index f163bc884..7e7c1cdbd 100644 --- a/services/core/tests/integration/runtime_wake_hint_integration_test.go +++ b/services/core/tests/integration/runtime_wake_hint_integration_test.go @@ -73,7 +73,7 @@ func newWakeHintIntegration(t *testing.T) *wakeHintIntegration { fakeCheckpointProvider: f.provider, sentinel: sentinel.owner.ID, release: make(chan struct{}), scans: make(chan int, 16), } - worker := startWebWorker(t, f.store, f.provider.registry, f.provider.link.Relay, f.key, provider, &f.policy) + worker := startWebWorker(t, f.store, f.provider.registry, f.provider.link.Relay, f.key, provider) onlineManagerNode(t, f.store, f.node) ctx, cancel := context.WithCancel(t.Context()) done := make(chan error, 1) diff --git a/services/core/tests/integration/sandbox_deployment_switch_worker_test.go b/services/core/tests/integration/sandbox_deployment_switch_worker_test.go index 7656e50ff..6c8d1635e 100644 --- a/services/core/tests/integration/sandbox_deployment_switch_worker_test.go +++ b/services/core/tests/integration/sandbox_deployment_switch_worker_test.go @@ -13,9 +13,9 @@ import ( "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sessions" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/deployment" - "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/execution" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/runtimegateway" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox" + "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox/providers" "github.com/google/uuid" ) @@ -26,20 +26,16 @@ func TestSandboxWorkerSwitchesAndRecoversFailedActivation(t *testing.T) { p := &lifecycleProvider{resources: map[string]sandbox.Info{}} var fail atomic.Bool var preparations atomic.Int32 - configuration := execution.NewDeferredRuntimeProvider(id, func(ctx context.Context) (*execution.RuntimeProvider, error) { - setup, err := deployments.Setup(ctx) - if err != nil || setup.Provider == "" { - return nil, err - } - return &execution.RuntimeProvider{InstallationID: id, ProviderKind: setup.Provider, Generation: setup.Generation, Mode: setup.Mode, SandboxLink: "wss://core.example/api/v1/sandbox-link", BackendFingerprint: setup.BackendFingerprint, Provider: p}, nil - }, func(ctx context.Context, setup deployment.Setup) (execution.PreparedRuntimeDeployment, error) { + registry := &fixtureProviderRegistry{t: t, provider: p, lookup: func(kind string) (providers.Adapter, error) { preparations.Add(1) if fail.Load() { - return execution.PreparedRuntimeDeployment{}, errors.New("fixture provider unavailable") + return providers.Adapter{}, errors.New("fixture provider unavailable") } - return execution.PreparedRuntimeDeployment{Config: &execution.RuntimeProvider{InstallationID: setup.InstallationID, ProviderKind: setup.Provider, Mode: setup.Mode, SandboxLink: "wss://core.example/api/v1/sandbox-link", BackendFingerprint: setup.BackendFingerprint, Provider: p}}, nil - }) - w := startWorker(t, t.Context(), s, &execution.Dispatcher{Registry: runtimegateway.NewRegistry(), ManagedRuntimes: configuration}) + return providers.Builtin().Lookup(kind) + }} + dispatcher := webDispatcher(t, id, p, runtimegateway.NewRegistry(), nil) + dispatcher.Providers = registry + w := startWorker(t, t.Context(), s, dispatcher) ctx, cancel := context.WithCancel(t.Context()) done := make(chan error, 1) go func() { done <- w.Run(ctx) }() diff --git a/services/core/tests/integration/sandbox_deployment_worker_test.go b/services/core/tests/integration/sandbox_deployment_worker_test.go index ff052e1c0..1f4653cee 100644 --- a/services/core/tests/integration/sandbox_deployment_worker_test.go +++ b/services/core/tests/integration/sandbox_deployment_worker_test.go @@ -22,19 +22,10 @@ func TestSandboxDeploymentWorkerActivatesWithoutRestart(t *testing.T) { deployments := deploymentService(t, s) id := uuid.NewString() p := &lifecycleProvider{resources: map[string]sandbox.Info{}} - configuration := execution.NewDeferredRuntimeProvider(id, func(ctx context.Context) (*execution.RuntimeProvider, error) { - setup, err := deployments.Setup(ctx) - if err != nil || setup.Provider == "" { - return nil, err - } - return &execution.RuntimeProvider{InstallationID: setup.InstallationID, ProviderKind: setup.Provider, Mode: setup.Mode, BackendFingerprint: setup.BackendFingerprint, SandboxLink: "wss://core.example/api/v1/sandbox-link", Provider: p}, nil - }, func(ctx context.Context, setup deployment.Setup) (execution.PreparedRuntimeDeployment, error) { - return execution.PreparedRuntimeDeployment{Config: &execution.RuntimeProvider{InstallationID: setup.InstallationID, ProviderKind: setup.Provider, Mode: setup.Mode, SandboxLink: "wss://core.example/api/v1/sandbox-link", BackendFingerprint: setup.BackendFingerprint, Provider: p}}, nil - }) start := func() (*execution.Worker, func()) { t.Helper() - w := startWorker(t, t.Context(), s, &execution.Dispatcher{Registry: runtimegateway.NewRegistry(), ManagedRuntimes: configuration}) + w := startWorker(t, t.Context(), s, webDispatcher(t, id, p, runtimegateway.NewRegistry(), nil)) var once sync.Once stop := func() { once.Do(func() { ctx, cancel := context.WithCancel(context.Background()); cancel(); _ = w.Run(ctx) }) diff --git a/services/core/tests/integration/worker_fixture_test.go b/services/core/tests/integration/worker_fixture_test.go index ba1134072..2d22be70d 100644 --- a/services/core/tests/integration/worker_fixture_test.go +++ b/services/core/tests/integration/worker_fixture_test.go @@ -2,6 +2,7 @@ package integration import ( "context" + "encoding/json" "errors" "os" "strings" @@ -11,13 +12,14 @@ import ( "github.com/google/uuid" "github.com/MiniMax-AI/OpenAgentCore/internal/sandboxlink/relay" - "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/deployment" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/execution" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/persistence/postgres/modelconfigurationpg" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/persistence/postgres/pgunit" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/persistence/postgres/sessionpg" + "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/providercontract" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/runtimegateway" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox" + "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox/providers" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sessions" ) @@ -28,7 +30,7 @@ import ( // observations through the model configuration adapter on s, runs Session use // cases and reads through the Session service and adapter on s, and reads the // deployment through the deployment adapter on s. Without the dispatcher's -// sandbox runtimes it runs fixtureRuntimes. +// provider dependencies it uses a lifecycleProvider fixture. func startWorker(t testing.TB, ctx context.Context, s *Store, dispatcher *execution.Dispatcher) *execution.Worker { t.Helper() worker, err := startWorkerErr(t, ctx, s, dispatcher) @@ -86,8 +88,23 @@ func startOwnedWorkerErr(t testing.TB, ctx context.Context, s *Store, dispatcher if owned.Links == nil { owned.Links = relay.New(runtimegateway.NewLinkAuthority(sessionAdapter(s))) } - if owned.ManagedRuntimes == nil { - owned.ManagedRuntimes = fixtureRuntimes(t, s) + if owned.InstallationID == "" { + view, err := deployments.View(ctx) + if err != nil { + return nil, errors.Join(err, owner.Lease.Close(ctx)) + } + owned.InstallationID = view.InstallationID + if owned.InstallationID == "" { + owned.InstallationID = testInstallation(t) + } + } + if owned.Providers == nil && owned.NodeProviders == nil { + p := &lifecycleProvider{resources: map[string]sandbox.Info{}} + owned.Providers = &fixtureProviderRegistry{t: t, provider: p, lookup: providers.Builtin().Lookup} + owned.NodeProviders = fixtureNodeProviders{t: t, provider: p} + } + if owned.SandboxLink == "" { + owned.SandboxLink = "wss://core.invalid/api/v1/sandbox-link" } return execution.StartWorker(ctx, &owned, owner) } @@ -104,22 +121,6 @@ func testInstallation(t testing.TB) string { return strings.TrimSpace(string(value)) } -// fixtureRuntimes is webRuntimes on a lifecycleProvider for the installation -// that claimed s's database, or for testInstallation before one did. On an -// unconfigured deployment it never loads a provider. -func fixtureRuntimes(t testing.TB, s *Store) *execution.RuntimeProvider { - t.Helper() - view, err := deploymentService(t, s).View(context.Background()) - if err != nil { - t.Fatal(err) - } - installation := view.InstallationID - if installation == "" { - installation = testInstallation(t) - } - return webRuntimes(t, s, installation, &lifecycleProvider{resources: map[string]sandbox.Info{}}, nil) -} - // executionOwner acquires the execution lease on s's database and builds the // execution operations on it, for tests that run them without a Worker. The // lease closes when the test ends. @@ -183,34 +184,64 @@ func webDeployment(t *testing.T, s *Store, provider string) string { return installation } -// webRuntimes is the Worker's sandbox runtimes as cmd/server builds them for -// installation: a deferred provider that runs s's committed Web setup on p. -func webRuntimes(t testing.TB, s *Store, installation string, p sandbox.SandboxProvider, suspension *execution.RuntimeSuspensionPolicy) *execution.RuntimeProvider { - deployments := deploymentService(t, s) - return execution.NewDeferredRuntimeProvider(installation, func(ctx context.Context) (*execution.RuntimeProvider, error) { - setup, err := deployments.Setup(ctx) - if err != nil || setup.Provider == "" { - return nil, err - } - return &execution.RuntimeProvider{InstallationID: setup.InstallationID, ProviderKind: setup.Provider, Mode: setup.Mode, Generation: setup.Generation, - SandboxLink: "wss://core.invalid/api/v1/sandbox-link", BackendFingerprint: setup.BackendFingerprint, Provider: p, Suspension: suspension}, nil - }, unusedPreparation(t)) +// fixtureProviderRegistry constructs the controlled provider through the same +// registry boundary as server startup. Setup declarations stay with the builtin registry. +type fixtureProviderRegistry struct { + t testing.TB + provider sandbox.SandboxProvider + lookup func(string) (providers.Adapter, error) } -// unusedPreparation is the preparer of a test that submits no sandbox -// selection through the Worker; preparing one fails the test. -func unusedPreparation(t testing.TB) execution.RuntimeDeploymentPreparer { - return func(context.Context, deployment.Setup) (execution.PreparedRuntimeDeployment, error) { - t.Error("the test prepared a sandbox selection it did not submit") - return execution.PreparedRuntimeDeployment{}, errors.New("unexpected sandbox selection preparation") +func (f *fixtureProviderRegistry) Lookup(kind string) (providers.Adapter, error) { + if f.lookup == nil { + f.t.Errorf("unexpected provider lookup for %q", kind) + return providers.Adapter{}, errors.New("unexpected provider lookup") } + return f.lookup(kind) +} +func (f *fixtureProviderRegistry) BuildDirect(sandbox.DirectConfig) (sandbox.SandboxProvider, error) { + if f.provider == nil { + f.t.Error("unexpected direct provider construction") + return nil, errors.New("unexpected direct provider construction") + } + return f.provider, nil +} +func (f *fixtureProviderRegistry) DiscoverSelection(_ context.Context, c sandbox.DirectConfig) (sandbox.Selection, error) { + return c.Selection, nil +} +func (f *fixtureProviderRegistry) DiscoverConfiguration(context.Context, string, sandbox.ConfigurationDiscoveryInput, sandbox.ProcessPaths) (json.RawMessage, error) { + f.t.Error("unexpected provider configuration discovery") + return nil, errors.New("unexpected provider configuration discovery") +} +func (f *fixtureProviderRegistry) VerifyCredential(context.Context, sandbox.DirectConfig, []sandbox.Reference) error { + f.t.Error("unexpected provider credential verification") + return errors.New("unexpected provider credential verification") +} + +type fixtureNodeProviders struct { + t testing.TB + provider sandbox.SandboxProvider +} + +func (f fixtureNodeProviders) Proxy(string, providercontract.Operations, uint64) sandbox.SandboxProvider { + if f.provider == nil { + f.t.Error("unexpected node provider construction") + return nil + } + return f.provider +} + +// webDispatcher uses the manager's committed setup loading and generation routing +// with controlled providers at the registry and node transport boundaries. +func webDispatcher(t testing.TB, installation string, p sandbox.SandboxProvider, registry *runtimegateway.Registry, links *relay.Relay) *execution.Dispatcher { + return &execution.Dispatcher{Registry: registry, Links: links, InstallationID: installation, SandboxLink: "wss://core.invalid/api/v1/sandbox-link", Providers: &fixtureProviderRegistry{t: t, provider: p, lookup: providers.Builtin().Lookup}, NodeProviders: fixtureNodeProviders{t: t, provider: p}} } -// startWebWorker starts the Worker on webRuntimes, with links as its Link -// relay or a new one when links is nil. -func startWebWorker(t *testing.T, s *Store, registry *runtimegateway.Registry, links *relay.Relay, installation string, p sandbox.SandboxProvider, suspension *execution.RuntimeSuspensionPolicy) *execution.Worker { +// startWebWorker starts the Worker on the committed deployment, with links as +// its Link relay or a new one when links is nil. +func startWebWorker(t *testing.T, s *Store, registry *runtimegateway.Registry, links *relay.Relay, installation string, p sandbox.SandboxProvider) *execution.Worker { t.Helper() - w, err := startNextWorker(t, t.Context(), s, &execution.Dispatcher{Registry: registry, Links: links, ManagedRuntimes: webRuntimes(t, s, installation, p, suspension)}) + w, err := startNextWorker(t, t.Context(), s, webDispatcher(t, installation, p, registry, links)) if err != nil { t.Fatal(err) }