fix: Various trace issues

[sergerad]

Summary

1. Adds instrumentation for the pre-validate_block part of sign block stack:

Fixed (acquire_permit and load_chain_tip):

rpc [ 595µs ] rpc.service: "validator.Api" | rpc.method: "SignBlock" ...
┝━ acquire_permit [ 2.92µs | 0.49% ]
┝━ load_chain_tip [ 103µs | 17.37% ]
┝━ validate_block [ 252µs | 42.38% ] tip.number: 0 | block.number: 1 ...
│  ┝━ sign_block [ 62.0µs ]
│  ┕━ store.block_store.save_block [ 28.8µs ]
┕━ persist_block_header [ 173µs | 29.12% ]

2. Fixes orphaned spans we are seeing on devnet:

Fixed (added tracing forest support to Validator for this):

sync_block [ 1.88ms ] block.number: 1
┕━ apply_block [ 1.88ms ]
   ┕━ write_block [ 1.88ms ] block.number: 1 ...
      ┝━ validate_block_header ...
      ┝━ apply_block (db) ... prune_history ...
      ┕━ apply_mutations ...

And sequencer side also:

block_builder.build_block [ 3.80ms | 0.00% / 100.00% ] block.number: 1 | block.batches.count: 0 | block.batch.ids: None | block.transactions.ids: None | block.transactions.count: 0 | block.updated_accounts.count: 0 | block.erased_note_proofs.count: 0 | block.nullifiers.count: 0 | block.output_notes.count: 0 | block.batches.output_notes.count: 0 | block.erased_notes.count: 0
┝━ block_builder.select_block [ 61.3µs | 0.64% / 1.61% ]
│  ┝━ mempool.lock [ 1.21µs | 0.03% ]
│  ┕━ mempool.select_block [ 35.6µs | 0.94% ] mempool.transactions.uncommitted: 0 | mempool.transactions.unbatched: 0 | mempool.batches.proposed: 0 | mempool.batches.proven: 0 | mempool.accounts: 0 | mempool.nullifiers: 0 | mempool.output_notes: 0
┝━ block_builder.get_block_inputs [ 280µs | 7.38% ]
┝━ block_builder.propose_block [ 30.4µs | 0.80% ]
┝━ block_builder.validate_block [ 1.10ms | 0.00% / 28.83% ]
│  ┕━ validator.client.validate_block [ 1.10ms | 28.83% ]
┕━ block_builder.commit_block [ 2.33ms | 0.00% / 61.37% ] block.number: 1 | block.commitment: 0xc9e804cffa4c7fe9def74ef4234a773aa78abd0342aea1b427f1940088266678 | block.transactions.count: 0
   ┝━ apply_block_with_proving_inputs [ 2.31ms | 0.00% / 60.74% ]
   │  ┝━ store.block_store.save_proving_inputs [ 22.1µs | 0.58% ] block.number: 1 | inputs_size: 654
   │  ┕━ apply_block [ 2.29ms | 0.00% / 60.16% ]
   │     ┕━ write_block [ 2.29ms | 35.64% / 60.16% ] block.number: 1 | block.commitment: 0xc9e804cffa4c7fe9def74ef4234a773aa78abd0342aea1b427f1940088266678 | block.transactions.count: 0 | snapshots.live: 1
   │        ┝━ validate_block_header [ 96.9µs | 2.55% ]
   ...

Each block is its own rooted sync_block tree instead of an endless chain under sync, and write_block is stitched under apply_block across the channel hop (previously it would have printed as a separate orphan tree). The sequencer side shows the same stitching under block_builder.commit_block → apply_block_with_proving_inputs → apply_block → write_block.

Changelog

[[entry]]
scope       = "validator"
impact      = "added"
description = "`decode` and `acquire_validation_permit` spans on the validator's sign-block and transaction submission paths."

[sergerad]

This might be unnecessary

[Mirko-von-Leipzig]

The function itself should be instrumented ideally

[sergerad]

We might prefer just to instrument the function declaration with the macro.

[kkovaacs]

Yep, we might actually want to instrument both upsert_block_header() and insert_validated_private_transaction() instead?

[Mirko-von-Leipzig]

Yes I think lets rather do that.

[kkovaacs]

Nice, thanks!

[kkovaacs]

Yep, we might actually want to instrument both upsert_block_header() and insert_validated_private_transaction() instead?

[Mirko-von-Leipzig]

I wonder if we shouldn't remove the feature, and instead make it a CLI toggle? Separate PR though :)

[Mirko-von-Leipzig]

Yes I think lets rather do that.

[Mirko-von-Leipzig]

The function itself should be instrumented ideally