Skip to content

feat(studio): add multidimensional comparison debugging - #888

Open
shenchengtsi wants to merge 2 commits into
volcengine:mainfrom
shenchengtsi:feat/multi-agent-comparison-debug
Open

feat(studio): add multidimensional comparison debugging#888
shenchengtsi wants to merge 2 commits into
volcengine:mainfrom
shenchengtsi:feat/multi-agent-comparison-debug

Conversation

@shenchengtsi

@shenchengtsi shenchengtsi commented Aug 13, 2026

Copy link
Copy Markdown

Summary

  • Add multidimensional comparison debugging for system prompt, model connection, and Skills across single- and multi-Agent drafts.
  • Show stable baseline-to-candidate differences, aligned traces, comparison history, and configuration details before starting a Session.
  • Replace stale comparison Sessions atomically: stage every new environment, keep prior evidence on failure, and clean old environments only after a successful swap.
  • Support transient debug model credentials and explicit Model ID, provider, and API Base overrides without persisting secrets.

Safety and limits

  • Limit each comparison to 4 active Runs and reserve up to 8 only during atomic Session replacement.
  • Require HTTPS for custom API Base values, an exact normalized hostname in VEADK_STUDIO_DEBUG_MODEL_HOST_ALLOWLIST, and the existing public-network validation.
  • Clear baseline and candidate temporary credentials after Cloud or Agent-topology changes, and mark the existing Session stale before it can be reused.
  • Abort and obsolete late SSE events by comparison round, Run, Session, and stream token.
  • Exclude rejected credential input from validation responses and keep stale transcripts read-only, including OAuth-backed actions, to prevent repeated external side effects.
  • Keep temporary API Keys in page memory and debug environment injection only; publishing requires credential confirmation again.

Verification

  • Frontend tests: 578 passed
  • TypeScript and production build: passed
  • Focused generated-agent backend tests: 89 passed
  • Full Python regression: 1546 passed, 5 skipped
  • Pre-commit: Ruff check, Ruff format, and hardcoded-secret scan passed
  • Independent scope and security review: approved with no remaining findings

The full Python regression ran before the final frontend-only credential invalidation fix; the frontend suite, TypeScript check, production build, and pre-commit checks were rerun afterward. The skipped Python cases require live external model or Pi environments, or apply only to legacy ADK behavior.

@shenchengtsi
shenchengtsi force-pushed the feat/multi-agent-comparison-debug branch from 4b543a7 to dae5e81 Compare August 13, 2026 07:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant