Problem
When streaming a desktop sandbox to spectators (users watching an AI agent work), the agent's mouse pointer is invisible on the stream.
Cause: getVNCCommand() starts x11vnc with default cursor handling, so the cursor is delivered via the cursor-shape pseudo-encoding. VNC clients render that shape at the viewer's own pointer position. A remote pointer moved programmatically (e.g. sandbox.moveMouse() / clicks from a computer-use agent) is therefore never drawn for a viewer who isn't moving their own mouse — which is exactly the spectator case that stream.start() + view-only embedding is used for.
For computer-use products this matters a lot: the whole point of the live stream is to watch the agent operate, and the pointer is the main thing to follow.
Proposal
Expose a cursor mode on the stream options, e.g.:
await sandbox.stream.start({ requireAuth: true, cursor: 'composite' })which appends -nocursorshape to the x11vnc command in getVNCCommand(). With that flag, x11vnc composites the cursor into the framebuffer updates, so every viewer sees the remote pointer at its true position, moving and clicking. Default stays 'shape' (current behavior).
It's a small change — the command is built in one place:
return `x11vnc -bg -display ${this.desktop.display} -forever -wait 50 -shared -rfbport ${this.vncPort} ${pwdFlag} ...`Current workaround
After stream.start(), we kill the SDK's x11vnc and re-run the same command with -nocursorshape appended. It works, but it duplicates SDK internals (display, port, auth flag) and will drift if those change — hence this request.
Happy to send a PR if you'd take one.
Problem
When streaming a desktop sandbox to spectators (users watching an AI agent work), the agent's mouse pointer is invisible on the stream.
Cause:
getVNCCommand()starts x11vnc with default cursor handling, so the cursor is delivered via the cursor-shape pseudo-encoding. VNC clients render that shape at the viewer's own pointer position. A remote pointer moved programmatically (e.g.sandbox.moveMouse()/ clicks from a computer-use agent) is therefore never drawn for a viewer who isn't moving their own mouse — which is exactly the spectator case thatstream.start()+ view-only embedding is used for.For computer-use products this matters a lot: the whole point of the live stream is to watch the agent operate, and the pointer is the main thing to follow.
Proposal
Expose a cursor mode on the stream options, e.g.:
which appends
-nocursorshapeto the x11vnc command ingetVNCCommand(). With that flag, x11vnc composites the cursor into the framebuffer updates, so every viewer sees the remote pointer at its true position, moving and clicking. Default stays'shape'(current behavior).It's a small change — the command is built in one place:
Current workaround
After
stream.start(), we kill the SDK's x11vnc and re-run the same command with-nocursorshapeappended. It works, but it duplicates SDK internals (display, port, auth flag) and will drift if those change — hence this request.Happy to send a PR if you'd take one.