Skip to content

Bind L2 compositional subtasks to the parent's instance - #160

Merged
aldro61 merged 2 commits into
mainfrom
fix/l2-subtask-instance-binding
Sep 23, 2026
Merged

aldro61 merged 2 commits into
mainfrom
fix/l2-subtask-instance-binding

Conversation

@aldro61

@aldro61 aldro61 commented Sep 23, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

CompositionalTask.setup_goal rebinds every subtask's instance and per-task user credentials to the parent's, but that loop sat inside the if self.level == 3: branch. At L2 it never ran, so any subtask built without the parent's instance kept its own SNowInstance(): an independent random pick from the instance pool, with admin credentials.

The browser acts on the parent's instance, but such a subtask validates and tears down on its own. Affected call sites include every NavigateAndOrder*Task (the order subtask gets the raw instance arg, i.e. None) and all four Filter{Asset,Hardware,Incident,User}ListTask subtasks in dash_do_filter.py (no instance= at all). Symptoms:

  • IndexError at tasks/list.py:939/:950: the filter subtask looks up the task's own agent sys_ids on the wrong instance and indexes an empty result.
  • HTTPError 404 deleting sc_request/<sys_id> in OrderHardwareTask.teardown: the request only exists on the parent's instance. This also hides the real failure, since the order validation had already looked on the wrong instance.
  • Orphaned records (per-task user, agents, incidents, assets, requests), because validation or teardown dies partway.

The fix runs the existing rebinding loop at every level, splitting the L3 block so the L3-only tasks are still appended before it and set up after it. One file, no per-call-site changes. No CompositionalTask is ever nested inside another (checked statically, including classes passed as *_class= and built later), so the parent always has its _base_user_* fields when the loop runs.

Why this went unnoticed

  • No CI job runs compositional tasks against the pool. unit_tests.yml excludes the pricy compositional tests on PRs, and its weekly job that does run them expects SNOW_INSTANCE_* secrets that aren't configured (they resolve to ''), so it has been failing for months without running anything. The daily pool workflow (instance_pool_ci.yml) runs only L1 tests, each given an explicit instance. The mismatch needs a multi-instance pool and unpinned construction, and gets more likely as instances are added.
  • Test retries mask it. test_cheat_compositional's @retry has no retry= predicate, so it retries on any exception (despite logging "Retrying due to a TimeoutError"), up to 5 times. Each retry redraws both instances, so a test only fails if every attempt mismatches: about 13% of affected tests on a 3-instance pool. Before this fix, the affected filter tasks were the slowest L2 tests (400–700s) because they were being retried.

Real benchmark runs are exposed as well: tasks are registered without task_kwargs and browsergym calls task_entrypoint(seed=seed), so any run against the pool that doesn't pass or pin an instance can grade L2 subtasks on the wrong instance.

Evidence

  • An instrumented run logged, for every failure, the parent's instance and the instance the failing call hit. They always differed (e.g. parent workarenapublic23, delete sent to workarenapublic25). The record was absent on the target instance and had no sys_audit_delete entry. For the filter failures, the missing sys_id was always the test's own agent.
  • Built exactly as the test builds them, 17/20 NavigateAndOrder* tasks bound parent and subtask to different instances on a 6-instance pool.
  • After the fix, a setup() check shows every subtask on the parent's instance and running as the per-task user.

Test plan

All runs unpinned, against the 6-instance pool, with -n 5 --slowmo 1000:

  • Previously failing families: 43/43 pass. That's every *FilterAssetListTask and NavigateAndOrder* L2 task, their infeasible variants, and 6 L3 regression checks. The same families failed 17/18 before the fix, run without retries.
  • Full L2 suite: 341/341 pass, up from 333 passed / 8 failed before. The other 4 pre-fix failures were the MacBook regression already reverted in Revert #158: restore original catalog item selector #159.
  • Full L3 suite: 341/341 pass (2h31m). Instances were being swapped in the pool during the run, so it covered both the old and the new pool. The retry harness still hides retries on this branch (Make CI actually run, and cover L2/L3 nightly #161 fixes that), but no test shows a retry-length outlier: the slowest are the inherently large tasks at 345–385s, the same as in the L2 sweeps.
  • New network-free regression test tests/test_compositional_binding.py: fails at L2 on the pre-fix code, passes with the fix.
  • black --check clean on the changed files.

Follow-ups (not in this PR)

  • Consider limiting test_cheat_compositional's retry to TimeoutError, as its log message implies. Retrying every exception hides nondeterministic bugs like this one, and each failed attempt leaves orphaned records.
  • Earlier runs against the pool left orphaned records: agent-style First.Last.NNNN users plus their incidents, assets and requests. These need a separate cleanup.

🤖 Generated with Claude Code

aldro61 and others added 2 commits September 23, 2026 12:07
CompositionalTask.setup_goal rebinds every subtask's instance and per-task
user credentials to the parent's, but that loop sat inside the level-3
branch. At L2, a subtask built without the parent's instance (e.g. the order
subtask in navigate_and_do.py, or the Filter*ListTask subtasks in
dash_do_filter.py) kept its own SNowInstance(), an independent random pick
from the instance pool with admin credentials. The browser acted on the
parent's instance while the subtask validated and tore down on another one,
causing IndexError in list validation and 404s deleting sc_request records,
and leaving orphaned records behind.

CI never saw it because it pins a single instance via SNOW_INSTANCE_URL.
Running the rebinding at every level fixes all affected L2 tasks at once.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Stubs subtasks that start bound to other instances, runs the real
CompositionalTask.setup_goal at L2 and L3, and checks every subtask ends up
on the parent's instance with the per-task user's credentials. Fails at L2 on
the pre-fix code; needs no ServiceNow instance, so it runs in the fast CI job.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@gitguardian-snow-gh1

Copy link
Copy Markdown

⚠️ GitGuardian has uncovered 3 secrets following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secrets in your pull request
GitGuardian id GitGuardian status Secret Commit Filename
51382 Triggered Generic Password df59370 tests/test_compositional_binding.py View secret
51383 Triggered Authentication Tuple df59370 tests/test_compositional_binding.py View secret
51383 Triggered Authentication Tuple df59370 tests/test_compositional_binding.py View secret
🛠 Guidelines to remediate hardcoded secrets

This is a non-blocking security notice.

A plain-text credential was located within this pull request. Before proceeding, you should understand the context of what this credential provides access to, and the implications of leaving the secret in-place.

Steps from here:

  1. Understand the implications of revoking this secret by investigating where it is used in your code.
  2. Replace and store your secret safely. Learn here the best practices.
  3. Revoke and rotate this secret.

To avoid such incidents in the future consider

Questions? Email devsecops@servicenow.com


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

@aldro61
aldro61 merged commit 302e20c into main Sep 23, 2026
5 of 8 checks passed
@aldro61
aldro61 deleted the fix/l2-subtask-instance-binding branch September 23, 2026 20:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant