Feat: Remote Workers Panel

[mickr777]

Summary

This PR explores optional multi-host rendering for InvokeAI v7. A primary InvokeAI instance can continue rendering locally while compatible workflows are dispatched to additional InvokeAI instances on the local network, without requiring a custom node pack or code changes on the workers.

Implementation

• Remote Workers panel: Configure additional InvokeAI instances, individual worker settings, optional per-user authentication, and automatic missing-model transfer.
• Four dispatch modes:
  • Mirror to all: Render locally and on every available configured remote worker.
  • Remotes only: Dispatch to remote workers; the primary runs the dispatch task rather than rendering the image locally.
  • Round-robin: Rotate generations between Local and available remote workers.
  • Auto-balance: Select the target with the fewest outstanding jobs tracked by the current browser's dispatch state. This is not a measurement of overall worker GPU utilization.
• Worker availability: Show Online, Offline, Login required, or Paused. Unavailable workers remain configured, are excluded from new dispatches, and can be used again when they return online.
• Credentials: Store each user's remote-worker credentials on the primary instance, without putting passwords in browser settings or workflow graphs.
• Previews and results: Show live Local and remote previews in Gallery and Canvas, including selectable Canvas staging previews. Import completed remote results into the selected Gallery or Canvas destination.
• Queue visibility: Display per-worker thumbnails, progress, and cancellation controls in the primary Queue. Associated remote work remains visible after the local queue item finishes.
• Cancellation: Cancel associated remote jobs from Gallery, Canvas, or the primary Queue, including after local rendering has finished. Cancellation also covers in-progress model transfers.
• Optional missing-model transfer: Transfer supported single-file models and complete Diffusers model directories to a remote, verify the installed model hash, and show transfer-progress notifications. Model transfer can be cancelled
• Optional and lightweight: Remote Workers should add minimal overhead to the InvokeAI installation and can be disabled at any time, allowing InvokeAI to operate normally with local generation only.
• Reuses existing InvokeAI APIs Builds on InvokeAI’s authenticated queue, image and model-management APIs rather than introducing a separate rendering system. New remote-worker and model-transfer endpoints remain subject to review

Related Issues / Discussions

I would particularly welcome feedback on whether multi-host rendering belongs in InvokeAI core, whether the queue/runtime integration is appropriate, and whether a different design would better fit the project.

QA Instructions

Manual test environment

• One Windows primary InvokeAI v7 instance, also rendering locally.
• One separate Unraid-hosted InvokeAI v7 linux Docker worker.
• One separate Windows systems running InvokeAI v7 as remote workers.
• One separate linux systems running InvokeAI v7 as remote workers.
• All three remote workers tested concurrently with the primary instance.
• Matching InvokeAI v7 builds and required models for the workflows under test.

Setup and checks

1. Run the primary and workers on matching InvokeAI v7 builds. No remote-worker code modifications or custom node pack are required. However the remote-workers will need host: 0.0.0.0 in there invoke.yaml file to allow network access
2. Configure workers in the Remote Workers panel. If a worker has multi-user authentication, save the relevant user's credentials on the primary.
3. Exercise the four dispatch modes, including Gallery and Canvas destinations, live previews, returned results, and cancellation after the local queue item completes.
4. Take a worker offline and bring it back online. Check that its availability changes without deleting and re-adding its configuration and that offline workers do not receive new dispatches.
5. For optional missing-model transfer, add allow_private_download_urls: true to each receiving worker's invoke.yaml. This is not needed if automatic model transfer is disabled. Test both a single-file model and a Diffusers directory, transfer-progress notifications, hash verification, and cancellation.

Known test limits / considerations

• The impact on a primary instance using multiple local GPUs has not been tested.
• Auto-balance uses browser-tracked outstanding dispatches, not the worker's complete system-wide queue or GPU load.
• Future database integration: Remote-worker configuration is currently stored in per-user browser storage, while saved credentials are encrypted on the primary instance. Storing per-user remote-worker configuration in InvokeAI’s database would allow settings to follow users across devices and provide server-side management. This is a potential future improvement and is outside the scope of this PR.
• Custom node compatibility: Remote workers must have compatible versions of any custom nodes required by a dispatched workflow. Missing or incompatible nodes may cause remote execution to fail. Custom node packs are not automatically transferred or installed.

Review

Compatibility / Rollout

Checklist

• The PR has a short but descriptive title, suitable for a changelog
• Meaningful regression coverage added / updated where needed; obsolete tests/code removed
• Persisted-state and API changes include required migrations / compatibility validation
• Relevant performance/efficiency opportunities considered; material claims have evidence
• Material review findings resolved and relevant checks rerun
• Documentation added / updated (if applicable)
• Updated What's New copy (if doing a release after this PR)

[lstein]

This is a big feature that has security implications. It will need a thorough review by multiple people. I am therefore marking it as DO NOT MERGE to avoid accidents!