fix: Bound tx validation and add spawn blocking calls

[sergerad]

Summary

Under load, the validator's SignBlock RPC was spiking from ~30ms to ~300ms, with the tracing added in #2487 showing the time spread uniformly across the request rather than concentrated in any one span — the signature of CPU starvation, not lock contention. Each submitted transaction runs proof verification and full VM re-execution on unbounded blocking threads, oversubscribing every core, while the Golden seal (secp256k1 ops) and the CPU-heavy parts of block signing ran directly on async workers.

The two paths have asymmetric failure modes: a delayed block signature stalls the whole chain, while a delayed transaction validation only slows admission — the RPC submits every transaction to every validator before it may enter the mempool, so blocks are proposed exclusively from already-validated transactions and signing never waits on validation. Signing must therefore always have CPU headroom; validation is the correct place to apply backpressure.

Changes:

• Bounded validation concurrency: SubmitProvenTransaction now acquires a permit from a semaphore sized to available_parallelism - 2 (minimum 1) before validating, converting CPU oversubscription into visible queueing. The permit is acquired after the already-validated short-circuit, so duplicate submissions never wait. The wait is instrumented as an acquire_validation_permit span.
• CPU work off the async workers: the Golden seal (secp256k1 group ops), into_header_and_body (account/nullifier/note tree roots plus chain and transaction commitments — hashing proportional to block contents), and full-block serialization now run via spawn_blocking_in_current_span, with panics propagated via resume_unwind (same convention as tx_validation).
• decode span: SignBlock::decode now times ProposedBlock deserialization, closing the last un-instrumented sync segment of the request path.

Verified against a local multi-validator run: the rpc span's self-time drops to ~0% (fully attributed), the offloaded work still reports under validate_block (the helper carries the span across the thread hop), and the uncontended acquire_validation_permit adds no measurable latency. Behavior note: under sustained validation overload, submissions now queue (and eventually hit the request timeout) instead of silently degrading block signing — backpressure lands on submitters, which already handle it.

Changelog

changelog = "none"
reason    = "Internal change only."

[kkovaacs]

Looks good to me % a small nit.

[Mirko-von-Leipzig]

Thank you; good findings. I initially thought it would be database write lock contention.

[Mirko-von-Leipzig]

@kkovaacs a fairly common pattern - should we make this part of the spawn function itself?

[Mirko-von-Leipzig]

This is one of the points where I wish we had more powerful tracing. It would be sensible to always have decode, encode and handle instrumented by the trait itself.

However that is not helpful because our macros need introspection of the function body to work properly.

Nothing actionable; just a complaint into the void :)

[Mirko-von-Leipzig]

Perhaps for a different PR, but I think this exhibits my point that most of our RPC work is actually not async, but rather blocking.

Our database calls are actually sync under the hood; we're just twisting ourselves into pretzels jumping between sync and async multiple times.