feat: add threadman worker for Node.js multi-threading support by Kolezhniuk · Pull Request #184 · iden3/ffjavascript

Kolezhniuk · 2026-03-31T14:10:18Z

Introduced threadman_worker.js to handle worker tasks using workerpool.
Updated utils.js to remove unnecessary global BigInt comment.
Refactored algebra tests to use Vitest framework, improving test structure and readability.
Converted all test files to use Vitest for assertions and lifecycle methods.
Added Vite configuration for building both Node.js and browser-compatible versions.
Implemented clean-up plugin in Vite to remove stale build artifacts before each build.
Enhanced test cases for various fields (Zq, Rat, Scalar) to ensure correctness and consistency.

- Introduced `threadman_worker.js` to handle worker tasks using workerpool. - Updated `utils.js` to remove unnecessary global BigInt comment. - Refactored algebra tests to use Vitest framework, improving test structure and readability. - Converted all test files to use Vitest for assertions and lifecycle methods. - Added Vite configuration for building both Node.js and browser-compatible versions. - Implemented clean-up plugin in Vite to remove stale build artifacts before each build. - Enhanced test cases for various fields (Zq, Rat, Scalar) to ensure correctness and consistency.

- Deleted `threadman_thread.cjs` and `threadman_worker.cjs` as part of the transition to ESM. - Updated `threadman.node.js` to use a build-time injected worker path instead of relying on `__dirname`. - Modified `vite.config.js` to inject `__BUILD_WORKER_PATH__` during the build process, ensuring correct worker path resolution.

… in engine_fft.js - Fixed the assignment of `pr` in `bn128.js` to use the correct property from `bn128wasmPrebuilt`. - Enhanced the FFT implementation in `engine_fft.js` to utilize a fast WASM path when input format matches the native stride, falling back to a JS implementation otherwise. - Updated `wasm_curve.js` to return a sliced copy of the input array in `toJacobian` when the byte length matches the expected size, ensuring immutability.

…mpatibility

- Implemented thread management logic in threadman_thread.cjs, including WebAssembly support for task execution. - Created thread function to handle memory allocation and task execution in a multi-threaded environment. - Added worker entry-point in threadman_worker.cjs to integrate with workerpool for Node.js. - Enabled task registration and execution within worker threads, ensuring compatibility with WebAssembly tasks.

…ript

…pt release

- Updated @emnapi/core and @emnapi/runtime to version 1.9.2 - Updated @oxc-project/types to version 0.124.0 - Updated @rolldown/binding packages to version 1.0.0-rc.15 - Updated vitest and related packages to version 4.1.4 - Updated globals to version 17.5.0 - Updated lru-cache to version 11.3.3 - Updated vite to version 8.0.8

Copilot

Pull request overview

This PR introduces a new workerpool-based threading implementation (“threadman worker”) to support multi-threaded execution across Node.js worker_threads and browser Web Workers, while also migrating the project’s build pipeline to Vite and the test suite to Vitest (including a browser test project via Playwright).

Changes:

Added platform adapters and a dedicated worker entrypoint to run threadman tasks via workerpool in Node and browsers.
Replaced Rollup configs with a unified vite.config.js that builds Node CJS plus browser ESM/IIFE bundles, and added ESLint flat config.
Migrated tests from Mocha/Chai to Vitest (with both Node and browser test projects) and updated CI accordingly.

Reviewed changes

Copilot reviewed 30 out of 37 changed files in this pull request and generated 15 comments.

Show a summary per file

File	Description
`vite.config.js`	Adds Vite build modes (node/browser/browser-iife), test projects, and build plugins for worker script inlining and cleanup.
`eslint.config.js`	Introduces ESLint flat config and project-wide formatting rules.
`package.json`	Updates exports/imports mapping for platform adapters; switches scripts to Vite/Vitest; changes dependencies.
`package-lock.json`	Reflects dependency and tooling migration (Vite/Vitest/Playwright/workerpool, etc.).
`src/threadman.js`	Refactors thread manager to use `workerpool` + platform adapter hooks and eager worker initialization.
`src/threadman.node.js`	Adds Node platform adapter (concurrency, worker type, worker source resolution).
`src/threadman.browser.js`	Adds browser platform adapter (Blob worker source using inlined worker script).
`src/threadman_worker.js`	New worker_threads entrypoint registering `runTask` with workerpool.
`src/threadman_thread.js`	Refactors worker task logic into a factory returning `runTask` for workerpool integration.
`src/bn128.js`	Updates wasmcurves import and gzip decode/decompression approach for prebuilt bn128 wasm.
`src/random.js`	Refactors RNG source selection (WebCrypto vs Node crypto fallback).
`src/engine_fft.js`	Adjusts reverse-permutation path selection (WASM vs JS) depending on element stride.
`src/engine_multiexp.js`	Formatting and small structural cleanups within multiexp task building.
`src/wasm_curve.js`	Changes `toJacobian` to return a copy when already in Jacobian form.
`src/utils.js`	Removes `/* global BigInt */` comment.
`src/scalar.js`	Removes `/* global BigInt */` comment.
`src/f1field.js`	Removes `/* global BigInt */` comment.
`src/polfield.js`	Minor comment indentation/formatting adjustment.
`test/algebra.js`	Migrates to Vitest APIs and updates lifecycle hooks (`beforeAll/afterAll`).
`test/bn128.js`	Migrates to Vitest and updates lifecycle hooks and formatting.
`test/pols.js`	Migrates to Vitest and removes Mocha-specific timeout usage.
`test/ratzqfield.js`	Migrates to Vitest and applies formatting cleanups.
`test/scalar.js`	Migrates to Vitest and updates suite/test naming.
`test/sqrt.js`	Migrates to Vitest lifecycle hooks and formatting cleanups.
`test/utils.js`	Migrates to Vitest and applies minor formatting fixes.
`test/zqfield.js`	Migrates to Vitest import style.
`.github/workflows/ci.yml`	Updates CI to use Node LTS on Ubuntu and installs Playwright for browser tests.
`build/threadman_worker.cjs`	Adds compiled CJS worker output (generated by Vite/Rolldown).
`build/threadman_thread.cjs`	Adds compiled CJS chunk for thread logic (generated by Vite/Rolldown).
`rollup.cjs.config.js`	Removes legacy Rollup config (Node build).
`rollup.browser.esm.config.js`	Removes legacy Rollup config (browser ESM build).
`.eslintrc.cjs`	Removes legacy ESLint config in favor of `eslint.config.js`.

Comments suppressed due to low confidence (1)

src/bn128.js:44

This decompression path requires atob, Blob, DecompressionStream, and Response.bytes(). Those Web APIs are not available in all supported Node.js runtimes, so buildBn128() may fail in Node. Add feature detection with a Node fallback (e.g., base64 decode via Buffer and gzip inflate via node:zlib) or keep the previous Node-compatible path.

        const compressedCode = Uint8Array.from(atob(bn128wasmPrebuilt.gzipCode), c => c.charCodeAt(0));
        const blob = new Blob([compressedCode]);

        const ds = new DecompressionStream("gzip");
        const decompressedStream = blob.stream().pipeThrough(ds);

        bn128wasm.code = await new Response(decompressedStream).bytes();
    } else {

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Kolezhniuk · 2026-06-04T16:08:07Z

+                    rollupOptions: {
+                        onwarn: suppressNodeExternalWarnings,
+                        output: { banner: "console.log('browser iife');" },
+                    },


Fixed — removed the console.log build banner from the browser IIFE config.

Kolezhniuk · 2026-06-04T16:08:09Z

+                rollupOptions: {
+                    external: isExternal,
+                    onwarn: suppressNodeExternalWarnings,
+                    output: { banner: "console.log('browser esm');" },
+                },


Fixed — removed the console.log banner from the browser ESM config.

Kolezhniuk · 2026-06-04T16:08:11Z

+                external: isExternal,
+                output: {
+                    banner: "console.log('node cjs');",
+                    // Stable chunk names so the require() path never changes
+                    // between rebuilds.
+                    chunkFileNames: "[name].cjs",
+                },


Fixed — removed the console.log banner from the Node CJS config.

Kolezhniuk · 2026-06-04T16:08:21Z

+        const raw = readFileSync(abs("src/threadman_thread.js"), "utf-8");
+        const fnSrc = raw.slice(raw.indexOf("export default ") + "export default ".length).trim();
+


Fixed — buildStart now throws a clear build error if the export default marker isn't found in threadman_thread.js, instead of silently slicing from index -1.

Kolezhniuk · 2026-06-04T16:08:22Z

+        // workerpool's pre-built, minified worker runtime (CJS string export).
+        const require = createRequire(import.meta.url);
+        const embeddedWorker = require("workerpool/src/generated/embeddedWorker");
+


Fixed — the require() of workerpool's internal embeddedWorker is now wrapped in try/catch and throws a clear, actionable error if that path moves between versions (with the original error as cause). workerpool exposes no public API for the embedded worker string, so we keep the internal path but now fail loudly.

Kolezhniuk · 2026-06-04T16:08:49Z

+on: [push, pull_request]

 jobs:
  test:
    name: Test
-    runs-on: ${{ matrix.os }}
-
-    strategy:
-      fail-fast: false
-      matrix:
-        os: [ubuntu-latest, macos-latest, windows-latest]
-        node-version: [lts/*, lts/-1, lts/-2]
+    timeout-minutes: 15
+    runs-on: ubuntu-latest

    steps:
      - name: Checkout project
-        uses: actions/checkout@v2
+        uses: actions/checkout@v6

      - name: Setup Node.js
-        uses: actions/setup-node@v3
+        uses: actions/setup-node@v6
        with:
-          node-version: ${{ matrix.node-version }}
-          check-latest: true
-          cache: 'npm'
+          node-version: "lts/*"
+          cache: "npm"


Intentional for now. The matrix was trimmed to keep the ESM-migration POC CI fast — the new browser/Playwright project adds significant time, and running it across Windows/macOS multiplies that. The OS/Node matrix should be restored before a real (non-POC) release.

Kolezhniuk · 2026-06-04T16:08:27Z

        run: npm ci

+      - name: Install Playwright dependencies
+        run: npx playwright install


Fixed — changed to npx playwright install --with-deps chromium so the required Linux system libraries are installed (only chromium is used in CI).

Kolezhniuk · 2026-06-04T16:08:50Z

    "wasmbuilder": "0.0.16",
-    "wasmcurves": "file:../wasmcurves",
-    "web-worker": "1.5.0"
+    "wasmcurves": "https://github.com/iden3/wasmcurves.git#feat/esm-migration",


Intentional and temporary. This points at the parallel feat/esm-migration branch of wasmcurves, which isn't released yet — the two migrations land together. It will be pinned to a published version/tag before merge.

Kolezhniuk · 2026-06-04T16:08:52Z

@@ -0,0 +1,22 @@
+console.log("node cjs");


Fixed at the source — removed the Node CJS banner in vite.config.js and regenerated the build, so this artifact no longer logs on load.

Kolezhniuk · 2026-06-04T16:08:54Z

@@ -0,0 +1,153 @@
+console.log("node cjs");


Fixed at the source — same banner removal + rebuild; this chunk no longer logs on require.

test(bn128): assert equality using Fr.eq for better clarity fix(scalar): correct test description for native conversion build(vite): enhance clean-node-build plugin to prevent unwanted deletions build(vite): improve error handling for worker script assembly

…iExp function

Kolezhniuk added 10 commits March 31, 2026 21:08

fix: replace rm -rf with rimraf in clean script for cross-platform co…

d53c632

…mpatibility

fix: update package-lock.json and reorder build scripts in package.json

df294ac

feat: migrate to ESM by updating module imports and adding prepare sc…

0e949b1

…ript

feat: enable headless mode for Playwright browser in Vite configuration

e9e3d4c

chore: update version to 0.3.1-poc in package.json for proof of conce…

1ff750d

…pt release

Kolezhniuk requested review from OBrezhniev, Copilot and vmidyllic and removed request for Copilot May 11, 2026 10:12

Kolezhniuk marked this pull request as ready for review May 11, 2026 10:12

Copilot started reviewing on behalf of Kolezhniuk May 11, 2026 10:12 View session

Copilot AI reviewed May 11, 2026

View reviewed changes

d5k9 added 2 commits June 5, 2026 09:14

refactor: optimize chunk size calculation and improve logging in mult…

c0ce9f6

…iExp function

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add threadman worker for Node.js multi-threading support#184

feat: add threadman worker for Node.js multi-threading support#184
Kolezhniuk wants to merge 12 commits into
feature/sharedArrayBuffersfrom
feat/esm-migration

Kolezhniuk commented Mar 31, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Kolezhniuk Jun 4, 2026

Uh oh!

Kolezhniuk Jun 4, 2026

Uh oh!

Kolezhniuk Jun 4, 2026

Uh oh!

Kolezhniuk Jun 4, 2026

Uh oh!

Kolezhniuk Jun 4, 2026

Uh oh!

Kolezhniuk Jun 4, 2026

Uh oh!

Kolezhniuk Jun 4, 2026

Uh oh!

Kolezhniuk Jun 4, 2026

Uh oh!

Kolezhniuk Jun 4, 2026

Uh oh!

Kolezhniuk Jun 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		const raw = readFileSync(abs("src/threadman_thread.js"), "utf-8");
		const fnSrc = raw.slice(raw.indexOf("export default ") + "export default ".length).trim();

Conversation

Kolezhniuk commented Mar 31, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants