Support _tidb_commit_ts for tiflash in next-gen and next-gen-columnar

[xzhangxian1008]

What problem does this PR solve?

Issue Number: ref #10733

Problem Summary:

What is changed and how it works?

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

None

Summary by CodeRabbit

• New Features

  • Improved disaggregated columnar reads involving hidden commit timestamps and MVCC version data.
  • Added support for late-materialization filters across projected and cast columns.
  • Improved handling of mixed column projections, generated columns, tombstones, and missing handles.

• Bug Fixes

  • Corrected commit-timestamp casting and filtering during columnar reads.
  • Fixed legacy reads to retrieve MVCC version values correctly.

• Tests

  • Added coverage for hidden commit timestamps, columnar wire formats, filter remapping, and mixed projections.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Advanced

Run ID: 2691017e-3659-40fa-968a-8964e35af886

📥 Commits

Reviewing files that changed from the base of the PR and between ad839fd and 48d90b6.

📒 Files selected for processing (6)

• dbms/src/Flash/Coprocessor/GenSchemaAndColumn.cpp
• dbms/src/Flash/Coprocessor/GenSchemaAndColumn.h
• dbms/src/Storages/StorageDisaggregated.cpp
• dbms/src/Storages/StorageDisaggregatedColumnar.cpp
• dbms/src/Storages/StorageDisaggregatedColumnar.h
• dbms/src/Storages/tests/gtest_hidden_commit_ts_column.cpp

Included review availability: Your plan provides up to 4 included reviews per hour; 3 remain after this review.

---

📝 Walkthrough

Walkthrough

Changes

Columnar commit-ts reads

Layer / File(s)	Summary
Shared columnar read generation dbms/src/Flash/Coprocessor/GenSchemaAndColumn.*	Adds shared helpers for storage column-ID mapping, table-scan metadata, table information, nullable version definitions, and late-materialization filter remapping.
Late-materialization integration dbms/src/Storages/StorageDisaggregated.cpp, dbms/src/Storages/StorageDisaggregatedColumnar.*	Columnar reads cast commit-ts values before predicate evaluation, use shared metadata generators, reset cast state, and read MVCC versions through fn_read_version.
Commit-ts behavior validation dbms/src/Storages/tests/gtest_hidden_commit_ts_column.cpp	Adds tests for disaggregated reads, columnar wire formats, filter remapping, casts, MVCC filtering, and mixed projections.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant ColumnarReader
  participant StorageMetadata
  participant MVCCVersionReader
  participant FilterEvaluator
  ColumnarReader->>StorageMetadata: Generate remapped scan and table metadata
  StorageMetadata-->>ColumnarReader: Return version-aware projection
  ColumnarReader->>MVCCVersionReader: Read nullable MVCC version values
  MVCCVersionReader-->>ColumnarReader: Return commit_ts values
  ColumnarReader->>FilterEvaluator: Cast values and evaluate remapped predicates
  FilterEvaluator-->>ColumnarReader: Return filtered block

Loading

Suggested reviewers: yongman

Merge Risk: ⚪ Minimal · up to 6aef1

No actionable merge-blocking issue was established for this change.

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The description follows the required template and identifies the issue, tests, side effects, documentation status, and release note. However, the required Problem Summary and explanation of what chang…	Complete the Problem Summary and the What is changed and how it works sections. Include the problem addressed, the implementation details for _tidb_commit_ts support, and the relevant test or manual-test steps.
Docstring Coverage	⚠️ Warning	Docstring coverage is 3.85% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 26 functions across 6 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the primary change: support for _tidb_commit_ts in TiFlash next-generation and next-generation-columnar modes.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

Full details: Description check

Explanation

The description follows the required template and identifies the issue, tests, side effects, documentation status, and release note. However, the required Problem Summary and explanation of what changed and how it works are empty.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{I’m a rabbit with columns to tend,
Commit-ts values now neatly bend.
Filters hop to their proper place,
Casts keep each version’s trace.
Tests thump softly: the paths now blend.}

_{Comment @coderabbitai help to get the list of available commands.}

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: coderabbitai[bot]
Once this PR has been reviewed and has the lgtm label, please assign guo-shaoge for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

@coderabbitai[bot]: adding LGTM is restricted to approvers and reviewers in OWNERS files.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[ti-chi-bot]

@xzhangxian1008: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-integration-next-gen	6aef1cd	link	true	/test pull-integration-next-gen
pull-integration-test	6aef1cd	link	true	/test pull-integration-test
pull-unit-test	6aef1cd	link	true	/test pull-unit-test
pull-sanitizer-asan	6aef1cd	link	false	/test pull-sanitizer-asan
pull-sanitizer-tsan	6aef1cd	link	false	/test pull-sanitizer-tsan

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.