feat(python-notebook-migration): add notebook-migration-service microservice in backend

[zyratlo]

What changes were proposed in this PR?

Introduces the microservice that mediates between Texera and the JupyterLab docker stack landed in migration-tool-jupyter-docker. Adds a new SBT subproject notebook-migration-service plus shared config and a frontend dev-proxy route.

New SBT subproject notebook-migration-service/:

• build.sbt and project/build.properties — module SBT setup; module depends on the existing Auth, Config, and DAO projects
• src/main/scala/.../NotebookMigrationService.scala — Dropwizard Application entry point; sets Jersey URL pattern to /api/*, registers the resource class, initializes the shared SQL connection via
  SqlServer.initConnection(StorageConfig.jdbcUrl, …), and wires in RequestLoggingFilter.
• src/main/scala/.../NotebookMigrationServiceConfiguration.scala — Dropwizard Configuration subclass.
• src/main/scala/.../resource/NotebookMigrationResource.scala — five REST endpoints under /notebook-migration:
  • GET /get-jupyter-url — health-checks the Jupyter container and returns its base URL.
  • GET /get-jupyter-iframe-url — returns the iframe-ready URL for notebook.ipynb.
  • POST /set-notebook — receives a notebook JSON, PUTs it into JupyterLab via its /api/contents/work/{name} API.
  • POST /store-notebook-and-mapping — persists a notebook + workflow-notebook mapping into Postgres in a single transaction (writes to the notebook and workflow_notebook_mapping tables added by migration-tool-database-tables).
  • POST /fetch-notebook-and-mapping — returns the most recent notebook + mapping for a given (wid, vid).
• src/main/resources/logback.xml — logging config.
• src/main/resources/notebook-migration-service-web-config.yaml — Dropwizard server config (HTTP port 9098, DB connection refs).

Root build wiring:

• build.sbt — declares the new NotebookMigrationService SBT subproject and adds it to the TexeraProject aggregation.

Shared config:

• common/config/src/main/resources/storage.conf — new jupyter { url = "http://localhost:9100" } block, overridable via STORAGE_JUPYTER_URL.
• common/config/src/main/scala/.../StorageConfig.scala — adds the jupyterURL accessor.

Frontend dev proxy:

• frontend/proxy.config.json — routes /api/notebook-migration/* to http://localhost:9098.

Any related issues, documentation, discussions?

Closes #5257
Parent issue #4301

• Hard dependency: must be merged after migration-tool-database-tables feat(python-notebook-migration): add database tables for Notebook Migration tool #5055 — the resource imports jOOQ-generated Notebook / WorkflowNotebookMapping classes that only exist once the schema PR is merged.
• Soft dependency: the JupyterLab container from migration-tool-jupyter-docker is what StorageConfig.jupyterURL points to. Without it running, the Jupyter-related endpoints return a 500 with "Cannot connect to Jupyter server". Service still starts and the DB-persistence endpoints work in isolation.

How was this PR tested?

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 4.7)

[codecov-commenter]

Codecov Report

❌ Patch coverage is 77.64706% with 38 lines in your changes missing coverage. Please review.
✅ Project coverage is 54.71%. Comparing base (dc33251) to head (a24161c).

Files with missing lines	Patch %	Lines
...ache/texera/service/NotebookMigrationService.scala	41.37%	17 Missing ⚠️
...a/service/resource/NotebookMigrationResource.scala	86.92%	2 Missing and 15 partials ⚠️
...rg/apache/texera/common/config/StorageConfig.scala	0.00%	2 Missing ⚠️
...ervice/NotebookMigrationServiceConfiguration.scala	0.00%	1 Missing ⚠️
...xera/service/resource/WorkflowAccessResource.scala	87.50%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #5258      +/-   ##
============================================
+ Coverage     54.64%   54.71%   +0.06%     
- Complexity     2922     2929       +7     
============================================
  Files          1109     1113       +4     
  Lines         42840    43010     +170     
  Branches       4608     4629      +21     
============================================
+ Hits          23412    23533     +121     
- Misses        18060    18088      +28     
- Partials       1368     1389      +21     

Flag	Coverage Δ		*Carryforward flag
access-control-service	70.44% <ø> (ø)
agent-service	34.36% <ø> (ø)
amber	56.67% <0.00%> (-0.02%)	⬇️
computing-unit-managing-service	1.65% <ø> (ø)
config-service	57.35% <ø> (ø)
file-service	58.59% <ø> (ø)
frontend	48.42% <ø> (-0.05%)	⬇️
notebook-migration-service	78.57% <78.57%> (?)
pyamber	90.20% <ø> (ø)
python	90.76% <ø> (ø)		Carriedforward from 439ea72
workflow-compiling-service	58.69% <ø> (ø)

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

⚠️ Benchmark changes need a look

🟢 0 better · 🔴 7 worse · ⚪ 8 noise (<±5%) · 0 without baseline

Compared against main dc33251 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

	config	throughput	MB/s	latency	max Δ latest / 7d
🔴	bs=10 sw=10 sl=64	395	0.241	24,786/36,810/36,810 us	🔴 +24.7% / 🔴 +146.1%
🔴	bs=100 sw=10 sl=64	822	0.501	118,129/168,841/168,841 us	🔴 +25.9% / 🔴 +68.5%
⚪	bs=1000 sw=10 sl=64	968	0.591	1,029,455/1,078,519/1,078,519 us	⚪ within ±5% / 🔴 +14.5%

Baseline details

Latest main dc33251 from same runner

config	metric	PR	latest main	7d avg	Δ latest	Δ 7d
bs=10 sw=10 sl=64	throughput	395 tuples/sec	456 tuples/sec	799.7 tuples/sec	-13.4%	-50.6%
bs=10 sw=10 sl=64	MB/s	0.241 MB/s	0.278 MB/s	0.488 MB/s	-13.3%	-50.6%
bs=10 sw=10 sl=64	p50	24,786 us	19,880 us	12,125 us	+24.7%	+104.4%
bs=10 sw=10 sl=64	p95	36,810 us	29,571 us	14,959 us	+24.5%	+146.1%
bs=10 sw=10 sl=64	p99	36,810 us	29,571 us	18,503 us	+24.5%	+98.9%
bs=100 sw=10 sl=64	throughput	822 tuples/sec	851 tuples/sec	1,069 tuples/sec	-3.4%	-23.1%
bs=100 sw=10 sl=64	MB/s	0.501 MB/s	0.52 MB/s	0.652 MB/s	-3.7%	-23.2%
bs=100 sw=10 sl=64	p50	118,129 us	116,062 us	92,709 us	+1.8%	+27.4%
bs=100 sw=10 sl=64	p95	168,841 us	134,112 us	100,227 us	+25.9%	+68.5%
bs=100 sw=10 sl=64	p99	168,841 us	134,112 us	109,877 us	+25.9%	+53.7%
bs=1000 sw=10 sl=64	throughput	968 tuples/sec	950 tuples/sec	1,114 tuples/sec	+1.9%	-13.1%
bs=1000 sw=10 sl=64	MB/s	0.591 MB/s	0.58 MB/s	0.68 MB/s	+1.9%	-13.1%
bs=1000 sw=10 sl=64	p50	1,029,455 us	1,051,715 us	899,197 us	-2.1%	+14.5%
bs=1000 sw=10 sl=64	p95	1,078,519 us	1,088,104 us	945,162 us	-0.9%	+14.1%
bs=1000 sw=10 sl=64	p99	1,078,519 us	1,088,104 us	986,231 us	-0.9%	+9.4%

Raw CSV

config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,506.80,200,128000,395,0.241,24786.12,36809.66,36809.66
1,100,10,64,20,2434.51,2000,1280000,822,0.501,118129.12,168841.43,168841.43
2,1000,10,64,20,20658.81,20000,12800000,968,0.591,1029455.16,1078519.38,1078519.38

[zyratlo]

/request-review @mengw15

[mengw15]

also please add more test cases

[github-actions]

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

• Contributors with relevant context: @Ma77Ball, @Yicong-Huang, @kunwp1
  You can notify them by mentioning @Ma77Ball, @Yicong-Huang, @kunwp1 in a comment.

[zyratlo]

also please add more test cases

Added more test cases, summary of leftover lines:

NotebookMigrationResource.scala — 100% line coverage. Every reachable branch is tested, including the Jupyter HTTP paths (via a stub HttpServer on the configured port), the write-access gate, and the malformed-input and unreachable-server failure paths. The only branches Codecov still flags as partial are unreachable by design:

• The NonFatal catch blocks — the missed branch is the fatal Throwable path that NonFatal deliberately does not catch.
• The @Auth resource-method wrappers — synthetic Scala/jacoco branches (parameter/entry artifacts), not real code paths.
• Object initialization and the conn == null guard in isJupyterAvailable (reached only if openConnection throws before assignment).

NotebookMigrationService.scala — run() and registerAuthFeatures are tested (mocked Dropwizard Environment, mirroring AccessControlServiceRunSpec). initialize() and main() are left untested: they stand up the real application and open the live JDBC connection, so they aren't unit-testable — consistent with the other Dropwizard services, which test run() only.

StorageConfig.scala / NotebookMigrationServiceConfiguration.scala — the flagged lines are config field declarations; their coverage is attributed to other modules' flags and can't be exercised from this service's tests.

[Copilot]

Pull request overview

Adds a new backend microservice (notebook-migration-service) to mediate between Texera and a JupyterLab stack for the Python Notebook Migration tool, including DB persistence for notebook + workflow mapping and wiring into the build/CI/dev-proxy.

Changes:

• Introduces notebook-migration-service Dropwizard app with notebook/Jupyter REST endpoints plus workflow write-access enforcement.
• Adds Scala tests covering DB transaction behavior, auth gating, and stubbed Jupyter success/failure paths.
• Wires the service into root sbt aggregation, CI platform matrix, shared StorageConfig, and the frontend dev proxy.

Reviewed changes

Copilot reviewed 16 out of 18 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
notebook-migration-service/src/main/scala/org/apache/texera/service/NotebookMigrationService.scala	Dropwizard application entrypoint; registers Jersey pattern, auth stack, resources, and request logging.
notebook-migration-service/src/main/scala/org/apache/texera/service/NotebookMigrationServiceConfiguration.scala	Dropwizard configuration class.
notebook-migration-service/src/main/scala/org/apache/texera/service/resource/NotebookMigrationResource.scala	Implements Jupyter connectivity + notebook upload, and DB store/fetch endpoints.
notebook-migration-service/src/main/scala/org/apache/texera/service/resource/WorkflowAccessResource.scala	Helper for checking workflow WRITE privilege.
notebook-migration-service/src/main/scala/org/apache/texera/service/resource/HealthCheckResource.scala	Adds a /healthcheck endpoint.
notebook-migration-service/src/test/scala/org/apache/texera/service/resource/NotebookMigrationResourceSpec.scala	Unit tests for DB persistence/rollback, access checks, and Jupyter stub interactions.
notebook-migration-service/src/test/scala/org/apache/texera/service/NotebookMigrationServiceRunSpec.scala	Verifies service run registers resources and auth features.
notebook-migration-service/src/main/resources/notebook-migration-service-web-config.yaml	Service port/logging config for Dropwizard.
notebook-migration-service/build.sbt	Service-specific build and dependencies.
notebook-migration-service/project/build.properties	Pins sbt version for the module build.
notebook-migration-service/LICENSE-binary	Binary distribution license manifest for dependency licensing checks.
notebook-migration-service/NOTICE-binary	Generated NOTICE aggregation for the service distribution.
common/config/src/main/resources/storage.conf	Adds storage.jupyter config (URL + token env overrides).
common/config/src/main/scala/org/apache/texera/common/config/StorageConfig.scala	Exposes Jupyter URL/token accessors.
frontend/proxy.config.json	Dev proxy route for /api/notebook-migration to localhost:9098.
build.sbt	Registers NotebookMigrationService as an sbt subproject and aggregates it.
bin/licensing/check_binary_deps.py	Includes the new service’s LICENSE-binary in the licensing check list.
.github/workflows/build.yml	Adds the new service to the platform CI matrix (dist + tests + licensing).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[mengw15]

LGTM. All earlier threads addressed, several with bonus improvements (the registerAuthFeatures helper with the anti-drift comment, the deliberately self-contained WorkflowAccessResource semantics, the Jackson-helper introduction for user-data JSON paths). Test coverage is solid — happy paths, transaction rollback on FK failure, 403 enforcement, fake-Jupyter stub for HTTP success/failure paths, name-validation rejection cases. Thanks for the iteration.

---

Forward-looking architectural note — not blocking this PR.

The per-user-pod deployment model used here (each user runs their own pod with both JupyterLab and a co-located notebook-migration-service JVM) is a reasonable v1 simplification: Jupyter URL is statically known within the pod, no routing layer is needed, and the @volatile var jupyterIframeURL singleton works because cross-user isolation is enforced at the pod boundary. The same-user-multi-tab case (two tabs of one user racing the singleton) is bounded — UX-only, since DB stays authoritative via store-notebook-and-mapping / fetch-notebook-and-mapping — and the comment in NotebookMigrationResource.scala already documents the cross-user limit.

The reason this is worth flagging at all is that Texera's other microservices follow a pretty consistent pattern:

Service	Deployment	Role
access-control-service	global	auth + Envoy ExtAuthz routing
file-service	global	stateless API over shared LakeFS
config-service	global	stateless config provider
workflow-compiling-service	global	stateless compilation
computing-unit-managing-service	global	orchestrator that routes to per-user compute pods
agent-service	global	stateless
compute / Jupyter pods	per-user	hold per-user kernel / notebook state

"Orchestrator services are global, stateful resources are per-user." computing-unit-managing-service is the closest analog — it's a global service that resolves per-user compute pods via the computing-unit-<cuid>.<pool>-svc.<namespace>.svc.cluster.local k8s DNS pattern (see checkComputingUnitAccess in AccessControlResource.scala). notebook-migration-service plays the same kind of orchestrator role for Jupyter pods, so over time it would be worth bringing into that pattern.

A rough migration path, staged so no single PR is too large:

1. API stateless refactor (backend-only, small): drop the @volatile var jupyterIframeURL, change getJupyterIframeURL to take notebookName as a parameter, remove the side-effect from setNotebook. Eliminates the multi-tab race even before any deployment change.
2. Per-user Jupyter URL resolution (backend + Helm): add jupyterUrlFor(uid) / jupyterTokenFor(uid) helpers backed by either k8s DNS lookup (mirroring computing-unit-managing-service) or a small registry table; spawn Jupyter pods per user with addressable k8s services.
3. Deployment topology (Helm only): move notebook-migration-service out of the per-user pod into a global Deployment; add NetworkPolicy to prevent cross-user Jupyter access.
4. Frontend coordination (the migration-tool series — #5260 and the upcoming UI PRs): frontend passes notebookName to getJupyterIframeURL, manages per-workflow naming, optionally adopts a wf<wid>_<name>.ipynb convention.

Open questions for when this gets picked up: k8s DNS pattern vs registry table for Jupyter URL resolution; per-pod tokens (more secure) vs shared (simpler); on-demand vs always-on Jupyter pod lifecycle. None of these need to be answered now — happy to defer the whole roadmap until the migration tool has shipped and real usage shows what scale / multi-tenancy issues actually surface. Just wanted to lay out the direction in one place so the per-user-pod choice doesn't get inherited by accident in later PRs.

Again: none of this blocks #5258 — v1 ships as-is. Approving.