Skip to content

Statistics::calculate_total_byte_size discards a known total_byte_size when num_rows is unknown #24026

Description

@bert-beyondloops

Describe the bug

Statistics::calculate_total_byte_size (datafusion/common/src/stats.rs:433) is meant to derive total_byte_size from num_rows and the schema's fixed-width columns. When all columns have a primitive width, it currently does:

  Some(size) => {                                                                                                                                                                                                                  
      self.total_byte_size = self.num_rows.multiply(&Precision::Exact(size)); 
 }     

Precision::multiply returns Precision::Absent whenever either operand is Absent. So if self.num_rows is Precision::Absent at the time this is called, total_byte_size is
overwritten with Precision::Absent — even if the statistics object already carried a perfectly good (exact or inexact) total_byte_size estimate computed some other way.

This silently throws away known size information any time row counts are unknown, which can degrade the quality of downstream statistics-based decisions.

The non-primitive-width branch already handles this correctly by downgrading the existing value with to_inexact() instead of discarding it — the primitive-width branch should do the same when num_rows is Absent.

To Reproduce

 use arrow::datatypes::{DataType, Field, Schema};                                                                                                                                                                                 
  use datafusion_common::stats::{Precision, Statistics};                                                                                                                                                                           
                                                                                                                                                                                                                                   
  let schema = Schema::new(vec![Field::new("a", DataType::Int32, false)]);                                                                                                                                                         
                                                                                                                                                                                                                                   
  let mut stats = Statistics::new_unknown(&schema);                                                                                                                                                                                
  stats.total_byte_size = Precision::Inexact(1234); // some previously known estimate                                                                                                                                              
  // num_rows is left as Precision::Absent                                                                                                                                                                                         
                                                                                                                                                                                                                                   
  stats.calculate_total_byte_size(&schema);                                                                                                                                                                                        
                                                                                                                                                                                                                                   
  // total_byte_size is now Precision::Absent, even though the schema is all                                                                                                                                                       
  // fixed-width and we had a prior estimate — expected Precision::Inexact(1234).                                                                                                                                                  
  assert_eq!(stats.total_byte_size, Precision::Absent);       

Expected behavior

When num_rows is Precision::Absent but the schema is entirely fixed-width, calculate_total_byte_size should preserve the previously-set total_byte_size (downgraded to inexact via to_inexact()), consistent with how the non-primitive-width case is handled, instead of discarding it to Absent.

Additional context

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions