fix(amber): surface region launch failures instead of hanging

[sshiv012]

What changes were proposed in this PR?

A worker failing to initialize in a region's second (non-dependee) phase, e.g. a Python UDF after a HashJoin, used to leave the execution RUNNING forever with no error. See the issue for the full chain.

• WorkflowExecutionManager.advanceRegionExecutions: keep the syncStatusAndTransitionRegionExecutionPhase() futures instead of dropping them in a foreach. The advance future now fails if any sync fails, so the existing .onFailure → FatalError in AdvanceRegionExecutionsHandler reports it. The rest of the method moved unchanged into advanceToNextRegions.
• RegionExecutionManager: a failed launch RPC (initializeExecutor, assignPort, openExecutor, startWorker) is wrapped as WorkflowRuntimeException with the target worker ID, so the FatalError names the failing worker and operator (review feedback).
• AsyncRPCClient.logControlReply: log the ControlError message, details and stack trace. It isn't a Throwable, so the logger silently dropped it.

	Before	After
UDF missing from pytexera import * after a join	hangs, all 0 tuples, no error	fails in ~1s with name 'UDFTableOperator' is not defined, attributed to the UDF worker
Log line	received error from …	received error from … (controlID: N): <message>

Behavior note: the future returned by advanceRegionExecutions now also waits on, and fails with, the syncs it triggers. AdvanceRegionExecutionsHandler doesn't wait on it. StartWorkflowHandler has no unfinished regions at start, so nothing changes there.

Any related issues, documentation, discussions?

Closes #8667

How was this PR tested?

Added tests. Each new failing-case test fails before the fix and passes after it.

• WorkflowExecutionManagerSpec, 2 new tests: a region whose probe op has a dependee port, followed by a UDF op.
  • The UDF's initializeExecutor returns a ControlError → the advance fails with that message.
  • The UDF initializes fine → it is started.
• DataProcessingSpec, 1 new e2e test on csv, csv → join → UDF: a Java UDF whose code doesn't compile, so the worker fails in initializeExecutor, the same step as a Python UDF with an import error.
  • Before the fix it times out after 1 min; after, it fails in ~0.5s, naming the UDF worker.
  • It uses a Java UDF because CI's amber job has no Python environment.
• RegionExecutionManagerTestSupport: CoordinatorRpcProbe takes an optional initializeExecutorResponse.

sbt 'WorkflowExecutionService/testOnly org.apache.texera.amber.engine.architecture.scheduling.WorkflowExecutionManagerSpec'
sbt 'WorkflowExecutionService/testOnly org.apache.texera.amber.engine.e2e.DataProcessingSpec -- -z "fail, not hang"'

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 5.5)

🤖 Generated with Claude Code

[github-actions]

Backport auto-label report

This fix: PR was checked against each actively-supported release branch. A release/* label nominates a backport target; the branch's release manager approving this PR is what sends the fix there. The required Backport Approvals check stays red until every label below is approved, so each manager either approves or removes their own label — which is why the labels left on a merged PR are exactly the branches it reached.

Release branch	Analysis
🚫 release/v1.3	Label was removed earlier (opt-out); not re-added. Re-add it by hand if this fix should be backported here after all — the branch's release manager then has to approve before this PR can merge.
🚫 release/v1.2	Label was removed earlier (opt-out); not re-added. Re-add it by hand if this fix should be backported here after all — the branch's release manager then has to approve before this PR can merge.

Auto-label run.

[github-actions]

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

• Contributors with relevant context: @Yicong-Huang, @aglinxinyuan, @aicam
  You can notify them by mentioning @Yicong-Huang, @aglinxinyuan, @aicam in a comment.

[github-actions]

⚠️ Benchmark changes need a look

🟢 2 better · 🔴 4 worse · ⚪ 9 noise (<±5%) · 0 without baseline

Compared against main 14de458 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

	config	throughput	MB/s	latency	max Δ latest / 7d
🟢	bs=10 sw=10 sl=64	404	0.247	23,734/30,374/30,374 us	🟢 -20.6% / 🔴 +80.2%
🔴	bs=100 sw=10 sl=64	797	0.486	120,098/190,042/190,042 us	🔴 +42.2% / 🔴 +69.5%
🔴	bs=1000 sw=10 sl=64	914	0.558	1,091,422/1,199,202/1,199,202 us	🔴 +5.8% / 🔴 +11.8%

Baseline details

Latest main 14de458 from same runner

config	metric	PR	latest main	7d avg	Δ latest	Δ 7d
bs=10 sw=10 sl=64	throughput	404 tuples/sec	403 tuples/sec	736.99 tuples/sec	+0.2%	-45.2%
bs=10 sw=10 sl=64	MB/s	0.247 MB/s	0.246 MB/s	0.45 MB/s	+0.4%	-45.1%
bs=10 sw=10 sl=64	p50	23,734 us	22,912 us	13,174 us	+3.6%	+80.2%
bs=10 sw=10 sl=64	p95	30,374 us	38,273 us	16,900 us	-20.6%	+79.7%
bs=10 sw=10 sl=64	p99	30,374 us	38,273 us	19,889 us	-20.6%	+52.7%
bs=100 sw=10 sl=64	throughput	797 tuples/sec	826 tuples/sec	946.47 tuples/sec	-3.5%	-15.8%
bs=100 sw=10 sl=64	MB/s	0.486 MB/s	0.504 MB/s	0.578 MB/s	-3.6%	-15.9%
bs=100 sw=10 sl=64	p50	120,098 us	121,032 us	105,257 us	-0.8%	+14.1%
bs=100 sw=10 sl=64	p95	190,042 us	133,653 us	112,150 us	+42.2%	+69.5%
bs=100 sw=10 sl=64	p99	190,042 us	133,653 us	125,962 us	+42.2%	+50.9%
bs=1000 sw=10 sl=64	throughput	914 tuples/sec	917 tuples/sec	973.64 tuples/sec	-0.3%	-6.1%
bs=1000 sw=10 sl=64	MB/s	0.558 MB/s	0.56 MB/s	0.594 MB/s	-0.4%	-6.1%
bs=1000 sw=10 sl=64	p50	1,091,422 us	1,086,193 us	1,032,217 us	+0.5%	+5.7%
bs=1000 sw=10 sl=64	p95	1,199,202 us	1,133,575 us	1,072,497 us	+5.8%	+11.8%
bs=1000 sw=10 sl=64	p99	1,199,202 us	1,133,575 us	1,102,623 us	+5.8%	+8.8%

Raw CSV

config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,494.98,200,128000,404,0.247,23734.03,30373.85,30373.85
1,100,10,64,20,2510.16,2000,1280000,797,0.486,120098.18,190042.25,190042.25
2,1000,10,64,20,21877.26,20000,12800000,914,0.558,1091422.25,1199201.68,1199201.68

[codecov-commenter]

Codecov Report

❌ Patch coverage is 80.00000% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.76%. Comparing base (14de458) to head (59a0fb9).
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
...chitecture/scheduling/RegionExecutionManager.scala	75.00%	0 Missing and 1 partial ⚠️
...exera/amber/engine/common/rpc/AsyncRPCClient.scala	66.66%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #8668      +/-   ##
============================================
+ Coverage     92.73%   92.76%   +0.02%     
+ Complexity     4943     4926      -17     
============================================
  Files          1242     1243       +1     
  Lines         52548    52451      -97     
  Branches       6492     6449      -43     
============================================
- Hits          48731    48656      -75     
+ Misses         2212     2194      -18     
+ Partials       1605     1601       -4     

Flag	Coverage Δ		*Carryforward flag
access-control-service	77.38% <ø> (ø)		Carriedforward from 577bc40
agent-service	99.16% <ø> (ø)		Carriedforward from 577bc40
amber	88.45% <80.00%> (-0.02%)	⬇️
computing-unit-managing-service	55.00% <ø> (-2.95%)	⬇️	Carriedforward from 577bc40
config-service	87.37% <ø> (ø)		Carriedforward from 577bc40
file-service	81.53% <ø> (ø)		Carriedforward from 577bc40
frontend	96.58% <ø> (+0.05%)	⬆️	Carriedforward from 577bc40
notebook-migration-service	83.73% <ø> (ø)		Carriedforward from 577bc40
pyamber	98.56% <ø> (-0.01%)	⬇️	Carriedforward from 577bc40
workflow-compiling-service	74.09% <ø> (ø)		Carriedforward from 577bc40

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[sshiv012]

/request-review @aicam

[xuang7]

Removed backport label: not a regression (the behavior predates v1.2), the failure is visible rather than silent.

[Ma77Ball]

@sshiv012 ty for the PR. Overall, this PR LGTM!

A minor nit is to consider carrying the failing worker ID into the surfaced FatalError found in AdvanceRegionExecutionsHandler.scala

[sshiv012]

@Ma77Ball Thanks, good call. Done in 1d583d5.

The failure arrived as a plain Throwable (rebuilt from the worker's ControlError), so relatedWorkerId was always None. Now each per-worker RPC that launches a region (initializeExecutor, assignPort, openExecutor, startWorker) wraps a failure as WorkflowRuntimeException(msg, cause, Some(workerId)). The existing FatalError(err, err.relatedWorkerId) then carries the worker ID, and the UI can attribute the error to that operator.

The tests now assert the worker ID too, and they failed before this change:

• unit: relatedWorkerId == Some(udfWorkerId)
• e2e: the Python UDF's worker ID

architecture.* 770/770, and scalafix and scalafmt are clean.

[Ma77Ball]

@sshiv012 there does appear to be a failing test in the DataProcessingSpec.scala. Please fix it before this can be merged.

[sshiv012]

Fixed in d784f5d. The new e2e tests started a real Python worker, which passed locally but not in CI, because CI's amber job has no Python environment?? The test now uses a Java UDF that fails to compile at the same setup step, so it needs no Python.