Core: Partition prune delete manifests in ManifestFilterManager by grantatspothero · Pull Request #18154 · apache/iceberg

grantatspothero · 2026-09-17T14:54:28Z

This PR: #13222 added support for tracking removed data files during MergingSnapshotProducer so that orphan DV could be removed.

The purpose of this PR is to increase performance of delete manifest filtering/rewriting through partition pruning delete manifests. See below commit breakdown:

2 commits in this PR, stacked on top of: #18197

Adds more parameters to the existing RewriteDataFilesBenchmark benchmark. I tried to make the benchmark more realistic (injecting object-storage like latency, using a partitioned table, etc).
Adds partition pruning into the delete manifest path. This mirrors the data manifest path.

JMH results:
commit (2)

Benchmark                                   (manifestTargetSizeBytes)  (metadataFileLatencyMs)  (numFiles)  (numPartitions)  (percentDataFilesRewritten)  Mode  Cnt   Score    Error  Units
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0       50000                0                            1    ss    5   0.240 ±  0.038   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0       50000                0                            5    ss    5   0.385 ±  0.192   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0       50000                0                           50    ss    5   1.071 ±  0.743   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0       50000                0                          100    ss    5   2.047 ±  1.011   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0       50000             1000                            1    ss    5   0.260 ±  0.092   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0       50000             1000                            5    ss    5   0.364 ±  0.103   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0       50000             1000                           50    ss    5   1.106 ±  0.568   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0       50000             1000                          100    ss    5   1.956 ±  0.658   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      100000                0                            1    ss    5   0.298 ±  0.143   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      100000                0                            5    ss    5   0.514 ±  0.184   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      100000                0                           50    ss    5   1.933 ±  0.873   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      100000                0                          100    ss    5   3.821 ±  3.269   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      100000             1000                            1    ss    5   0.302 ±  0.143   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      100000             1000                            5    ss    5   0.540 ±  0.276   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      100000             1000                           50    ss    5   1.905 ±  0.947   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      100000             1000                          100    ss    5   3.650 ±  2.995   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      500000                0                            1    ss    5   0.559 ±  0.203   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      500000                0                            5    ss    5   1.326 ±  1.889   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      500000                0                           50    ss    5  10.509 ±  5.856   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      500000                0                          100    ss    5  27.937 ± 33.988   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      500000             1000                            1    ss    5   0.729 ±  0.422   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      500000             1000                            5    ss    5   1.607 ±  0.531   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      500000             1000                           50    ss    5  12.706 ± 20.017   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      500000             1000                          100    ss    5  29.234 ± 33.748   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50       50000                0                            1    ss    5   0.787 ±  0.130   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50       50000                0                            5    ss    5   1.119 ±  0.119   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50       50000                0                           50    ss    5   2.921 ±  0.206   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50       50000                0                          100    ss    5   4.449 ±  0.259   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50       50000             1000                            1    ss    5   0.762 ±  0.064   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50       50000             1000                            5    ss    5   1.114 ±  0.274   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50       50000             1000                           50    ss    5   2.922 ±  0.181   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50       50000             1000                          100    ss    5   4.546 ±  0.578   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      100000                0                            1    ss    5   0.996 ±  0.079   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      100000                0                            5    ss    5   1.575 ±  0.107   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      100000                0                           50    ss    5   4.382 ±  0.248   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      100000                0                          100    ss    5   7.181 ±  1.516   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      100000             1000                            1    ss    5   1.000 ±  0.085   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      100000             1000                            5    ss    5   1.584 ±  0.097   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      100000             1000                           50    ss    5   4.396 ±  0.251   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      100000             1000                          100    ss    5   7.647 ±  2.419   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      500000                0                            1    ss    5   2.791 ±  0.072   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      500000                0                            5    ss    5   4.353 ±  0.516   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      500000                0                           50    ss    5  18.207 ±  3.437   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      500000                0                          100    ss    5  37.054 ± 21.876   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      500000             1000                            1    ss    5   2.869 ±  0.281   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      500000             1000                            5    ss    5   4.403 ±  0.355   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      500000             1000                           50    ss    5  22.089 ±  9.031   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      500000             1000                          100    ss    5  40.231 ± 45.683   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0       50000                0                            1    ss    5   0.560 ±  0.263   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0       50000                0                            5    ss    5   0.524 ±  0.158   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0       50000                0                           50    ss    5   1.979 ±  0.469   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0       50000                0                          100    ss    5   2.146 ±  0.093   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0       50000             1000                            1    ss    5   0.475 ±  0.059   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0       50000             1000                            5    ss    5   0.511 ±  0.033   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0       50000             1000                           50    ss    5   1.950 ±  0.083   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0       50000             1000                          100    ss    5   2.269 ±  0.300   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      100000                0                            1    ss    5   0.546 ±  0.345   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      100000                0                            5    ss    5   0.553 ±  0.193   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      100000                0                           50    ss    5   2.216 ±  0.354   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      100000                0                          100    ss    5   2.561 ±  0.239   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      100000             1000                            1    ss    5   0.478 ±  0.082   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      100000             1000                            5    ss    5   0.637 ±  0.911   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      100000             1000                           50    ss    5   2.456 ±  1.072   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      100000             1000                          100    ss    5   2.931 ±  1.298   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      500000                0                            1    ss    5   1.139 ±  0.323   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      500000                0                            5    ss    5   1.237 ±  0.494   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      500000                0                           50    ss    5   5.043 ±  1.615   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      500000                0                          100    ss    5   7.641 ±  4.469   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      500000             1000                            1    ss    5   1.243 ±  0.533   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      500000             1000                            5    ss    5   1.537 ±  0.607   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      500000             1000                           50    ss    5   6.671 ±  2.818   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      500000             1000                          100    ss    5   6.675 ±  1.322   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50       50000                0                            1    ss    5   0.968 ±  0.101   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50       50000                0                            5    ss    5   1.226 ±  0.536   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50       50000                0                           50    ss    5   2.720 ±  0.199   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50       50000                0                          100    ss    5   2.920 ±  0.263   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50       50000             1000                            1    ss    5   0.941 ±  0.036   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50       50000             1000                            5    ss    5   0.983 ±  0.125   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50       50000             1000                           50    ss    5   2.734 ±  0.118   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50       50000             1000                          100    ss    5   3.022 ±  0.440   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      100000                0                            1    ss    5   1.091 ±  0.341   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      100000                0                            5    ss    5   1.032 ±  0.082   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      100000                0                           50    ss    5   3.110 ±  0.353   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      100000                0                          100    ss    5   3.635 ±  0.315   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      100000             1000                            1    ss    5   0.993 ±  0.096   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      100000             1000                            5    ss    5   1.034 ±  0.036   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      100000             1000                           50    ss    5   3.090 ±  0.289   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      100000             1000                          100    ss    5   3.830 ±  0.522   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      500000                0                            1    ss    5   1.624 ±  0.088   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      500000                0                            5    ss    5   1.868 ±  0.790   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      500000                0                           50    ss    5   6.796 ±  2.851   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      500000                0                          100    ss    5   8.207 ±  2.527   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      500000             1000                            1    ss    5   1.708 ±  0.576   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      500000             1000                            5    ss    5   2.426 ±  1.983   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      500000             1000                           50    ss    5   7.736 ±  2.159   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      500000             1000                          100    ss    5   7.131 ±  0.931   s/op

commit (3):

RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0       50000                0                            1    ss    5   0.247 ± 0.070   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0       50000                0                            5    ss    5   0.379 ± 0.181   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0       50000                0                           50    ss    5   1.026 ± 0.132   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0       50000                0                          100    ss    5   1.924 ± 0.259   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0       50000             1000                            1    ss    5   0.220 ± 0.047   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0       50000             1000                            5    ss    5   0.343 ± 0.149   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0       50000             1000                           50    ss    5   1.023 ± 0.168   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0       50000             1000                          100    ss    5   1.862 ± 0.259   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      100000                0                            1    ss    5   0.282 ± 0.144   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      100000                0                            5    ss    5   0.473 ± 0.224   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      100000                0                           50    ss    5   1.787 ± 0.109   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      100000                0                          100    ss    5   3.267 ± 0.231   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      100000             1000                            1    ss    5   0.279 ± 0.126   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      100000             1000                            5    ss    5   0.511 ± 0.335   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      100000             1000                           50    ss    5   1.830 ± 0.364   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      100000             1000                          100    ss    5   3.235 ± 0.116   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      500000                0                            1    ss    5   0.613 ± 0.162   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      500000                0                            5    ss    5   1.217 ± 0.557   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      500000                0                           50    ss    5   9.368 ± 1.900   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      500000                0                          100    ss    5  24.045 ± 7.809   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      500000             1000                            1    ss    5   0.861 ± 0.281   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      500000             1000                            5    ss    5   1.388 ± 2.192   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      500000             1000                           50    ss    5  12.689 ± 1.326   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                        0      500000             1000                          100    ss    5  24.632 ± 4.927   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50       50000                0                            1    ss    5   0.799 ± 0.072   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50       50000                0                            5    ss    5   1.080 ± 0.102   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50       50000                0                           50    ss    5   3.106 ± 0.506   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50       50000                0                          100    ss    5   4.482 ± 1.094   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50       50000             1000                            1    ss    5   0.754 ± 0.064   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50       50000             1000                            5    ss    5   1.026 ± 0.037   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50       50000             1000                           50    ss    5   2.921 ± 0.174   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50       50000             1000                          100    ss    5   4.527 ± 1.071   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      100000                0                            1    ss    5   0.993 ± 0.020   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      100000                0                            5    ss    5   1.597 ± 0.084   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      100000                0                           50    ss    5   4.435 ± 0.549   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      100000                0                          100    ss    5   6.963 ± 1.439   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      100000             1000                            1    ss    5   0.833 ± 0.088   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      100000             1000                            5    ss    5   1.464 ± 0.224   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      100000             1000                           50    ss    5   4.440 ± 0.700   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      100000             1000                          100    ss    5   7.265 ± 1.029   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      500000                0                            1    ss    5   2.913 ± 0.574   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      500000                0                            5    ss    5   4.400 ± 0.444   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      500000                0                           50    ss    5  17.025 ± 2.188   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      500000                0                          100    ss    5  34.733 ± 2.459   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      500000             1000                            1    ss    5   1.658 ± 0.345   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      500000             1000                            5    ss    5   3.183 ± 0.533   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      500000             1000                           50    ss    5  18.244 ± 1.344   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      500000             1000                          100    ss    5  36.146 ± 4.005   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0       50000                0                            1    ss    5   0.487 ± 0.062   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0       50000                0                            5    ss    5   0.508 ± 0.060   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0       50000                0                           50    ss    5   2.003 ± 0.051   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0       50000                0                          100    ss    5   2.584 ± 0.444   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0       50000             1000                            1    ss    5   0.476 ± 0.043   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0       50000             1000                            5    ss    5   0.509 ± 0.088   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0       50000             1000                           50    ss    5   1.939 ± 0.110   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0       50000             1000                          100    ss    5   2.210 ± 0.191   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      100000                0                            1    ss    5   0.504 ± 0.228   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      100000                0                            5    ss    5   0.568 ± 0.275   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      100000                0                           50    ss    5   2.271 ± 0.532   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      100000                0                          100    ss    5   2.718 ± 0.533   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      100000             1000                            1    ss    5   0.591 ± 0.462   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      100000             1000                            5    ss    5   0.508 ± 0.047   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      100000             1000                           50    ss    5   2.390 ± 0.840   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      100000             1000                          100    ss    5   2.796 ± 0.898   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      500000                0                            1    ss    5   1.046 ± 0.094   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      500000                0                            5    ss    5   1.246 ± 0.281   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      500000                0                           50    ss    5   4.596 ± 1.478   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      500000                0                          100    ss    5   7.689 ± 2.578   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      500000             1000                            1    ss    5   1.337 ± 1.510   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      500000             1000                            5    ss    5   1.424 ± 0.758   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      500000             1000                           50    ss    5   5.610 ± 2.835   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                        0      500000             1000                          100    ss    5  10.338 ± 3.360   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50       50000                0                            1    ss    5   1.176 ± 1.446   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50       50000                0                            5    ss    5   1.102 ± 0.626   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50       50000                0                           50    ss    5   2.982 ± 0.515   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50       50000                0                          100    ss    5   2.986 ± 0.499   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50       50000             1000                            1    ss    5   1.039 ± 0.501   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50       50000             1000                            5    ss    5   0.964 ± 0.059   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50       50000             1000                           50    ss    5   2.725 ± 0.139   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50       50000             1000                          100    ss    5   2.971 ± 0.389   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      100000                0                            1    ss    5   0.992 ± 0.228   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      100000                0                            5    ss    5   0.977 ± 0.033   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      100000                0                           50    ss    5   3.017 ± 0.137   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      100000                0                          100    ss    5   3.706 ± 1.082   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      100000             1000                            1    ss    5   0.973 ± 0.082   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      100000             1000                            5    ss    5   1.033 ± 0.051   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      100000             1000                           50    ss    5   3.118 ± 0.291   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      100000             1000                          100    ss    5   3.638 ± 0.398   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      500000                0                            1    ss    5   1.562 ± 0.116   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      500000                0                            5    ss    5   1.854 ± 1.245   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      500000                0                           50    ss    5   5.304 ± 1.084   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      500000                0                          100    ss    5   9.529 ± 3.073   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      500000             1000                            1    ss    5   1.694 ± 0.341   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      500000             1000                            5    ss    5   2.154 ± 1.048   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      500000             1000                           50    ss    5   6.544 ± 3.243   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                    8388608                       50      500000             1000                          100    ss    5  11.614 ± 8.374   s/op

Most of the benchmark shows no change as expected, the notable improvements are for the "large partitioned table with realistic object storage latency" cases:

(2)

RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      500000             1000                            1    ss    5   2.869 ±  0.281   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      500000             1000                            5    ss    5   4.403 ±  0.355   s/op

(3):

RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      500000             1000                            1    ss    5   1.658 ± 0.345   s/op
RewriteDataFilesBenchmark.rewriteDataFiles                      65536                       50      500000             1000                            5    ss    5   3.183 ± 0.533   s/op

grantatspothero · 2026-09-18T17:41:25Z

cc: @nastra since you built the referencing PR.

nastra · 2026-09-21T10:31:05Z

@grantatspothero can you please add some details about the bug that is being fixed here? Also it sounds like the benchmark changes are independent from the bugfix, so it would be better to do those in a separate PR.
Generally speaking, it's best to not mix different things into the PR if possible, because that makes reviewing more difficult and increases the time until a change can get merged.

Orphaned DVs were sometimes never removed in ManifestFilterManager. canContainDroppedFiles only returned true for a delete manifest whose minSequenceNumber was below the dropDeleteFilesOlderThan threshold when data files were also being removed in the same commit. A commit that only calls dropDeleteFilesOlderThan (e.g. a plain append) never admitted delete manifests for that check.

The deletePaths/deleteFiles/removedDataFilePaths checks were chained as else-if branches, so once one of the earlier conditions applied, later ones were never evaluated. This meant a manifest could be skipped (and a dangling DV missed) whenever both deleteFiles and removedDataFilePaths were non-empty

…ataFilesBenchmark The benchmark builds an unpartitioned table on the local filesystem, so it cannot show the cost of reading or writing manifests. A local open takes well under a millisecond while an object store charges tens of milliseconds per round trip, and manifest work against remote storage is dominated by those round trips rather than by decoding.

Removing data files makes every DV that references them dangling, so ManifestFilterManager scans delete manifests to find them. Previously canContainDroppedFiles admitted every delete manifest unconditionally, so a commit that replaces data files opened every delete manifest. Now uses partition pruning to determine which delete manifests to open. This mirrors partition pruning done with data manifests.

grantatspothero · 2026-09-21T14:09:57Z

Split out the fix commits from the partition pushdown performance improvement commit. This PR is now stacked on top of the bug fix PR: #18197

github-actions Bot added the core label Sep 17, 2026

grantatspothero force-pushed the gn/pruneDeleteManifestScanForDanglingDVs branch 2 times, most recently from e80b3f8 to ffb5a5a Compare September 17, 2026 15:21

grantatspothero changed the title ~~Partition prune delete manifests in ManifestFilterManager~~ Core: Partition prune delete manifests in ManifestFilterManager Sep 17, 2026

grantatspothero force-pushed the gn/pruneDeleteManifestScanForDanglingDVs branch 3 times, most recently from 634fca2 to 6dca9e2 Compare September 18, 2026 17:34

grantatspothero added 4 commits September 21, 2026 08:50

grantatspothero force-pushed the gn/pruneDeleteManifestScanForDanglingDVs branch from 6dca9e2 to b247cb4 Compare September 21, 2026 14:08

Navigation Menu

Core: Partition prune delete manifests in ManifestFilterManager - #18154

Core: Partition prune delete manifests in ManifestFilterManager#18154
grantatspothero wants to merge 4 commits into
apache:mainfrom
grantatspothero:gn/pruneDeleteManifestScanForDanglingDVs

grantatspothero commented Sep 17, 2026 •

edited

Loading

Uh oh!

grantatspothero commented Sep 18, 2026

Uh oh!

nastra commented Sep 21, 2026

Uh oh!

grantatspothero commented Sep 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

grantatspothero commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

grantatspothero commented Sep 18, 2026

Uh oh!

nastra commented Sep 21, 2026

Uh oh!

grantatspothero commented Sep 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

grantatspothero commented Sep 17, 2026 •

edited

Loading