Describe the bug
Statistics::calculate_total_byte_size (datafusion/common/src/stats.rs:433) is meant to derive total_byte_size from num_rows and the schema's fixed-width columns. When all columns have a primitive width, it currently does:
Some(size) => {
self.total_byte_size = self.num_rows.multiply(&Precision::Exact(size));
}
Precision::multiply returns Precision::Absent whenever either operand is Absent. So if self.num_rows is Precision::Absent at the time this is called, total_byte_size is
overwritten with Precision::Absent — even if the statistics object already carried a perfectly good (exact or inexact) total_byte_size estimate computed some other way.
This silently throws away known size information any time row counts are unknown, which can degrade the quality of downstream statistics-based decisions.
The non-primitive-width branch already handles this correctly by downgrading the existing value with to_inexact() instead of discarding it — the primitive-width branch should do the same when num_rows is Absent.
To Reproduce
use arrow::datatypes::{DataType, Field, Schema};
use datafusion_common::stats::{Precision, Statistics};
let schema = Schema::new(vec![Field::new("a", DataType::Int32, false)]);
let mut stats = Statistics::new_unknown(&schema);
stats.total_byte_size = Precision::Inexact(1234); // some previously known estimate
// num_rows is left as Precision::Absent
stats.calculate_total_byte_size(&schema);
// total_byte_size is now Precision::Absent, even though the schema is all
// fixed-width and we had a prior estimate — expected Precision::Inexact(1234).
assert_eq!(stats.total_byte_size, Precision::Absent);
Expected behavior
When num_rows is Precision::Absent but the schema is entirely fixed-width, calculate_total_byte_size should preserve the previously-set total_byte_size (downgraded to inexact via to_inexact()), consistent with how the non-primitive-width case is handled, instead of discarding it to Absent.
Additional context
No response
Describe the bug
Statistics::calculate_total_byte_size(datafusion/common/src/stats.rs:433) is meant to derive total_byte_size from num_rows and the schema's fixed-width columns. When all columns have a primitive width, it currently does:Precision::multiplyreturnsPrecision::Absentwhenever either operand isAbsent. So ifself.num_rowsisPrecision::Absentat the time this is called,total_byte_sizeisoverwritten with
Precision::Absent— even if the statistics object already carried a perfectly good (exact or inexact)total_byte_sizeestimate computed some other way.This silently throws away known size information any time row counts are unknown, which can degrade the quality of downstream statistics-based decisions.
The non-primitive-width branch already handles this correctly by downgrading the existing value with
to_inexact()instead of discarding it — the primitive-width branch should do the same whennum_rowsisAbsent.To Reproduce
Expected behavior
When
num_rowsisPrecision::Absentbut the schema is entirely fixed-width,calculate_total_byte_sizeshould preserve the previously-settotal_byte_size(downgraded to inexact viato_inexact()), consistent with how the non-primitive-width case is handled, instead of discarding it toAbsent.Additional context
No response