khiops.sklearn : fit() method : bad management of date type columns

[DTRGit]

Description

The Error "'bool' object has no attribute 'to_numpy'" is raised when I have a date type column in my table.

Questions/Ideas

A potential workaround is to CAST the date column into a STRING type column

Context

• Khiops version

Khiops 11.0.1

Environment variables:
KHIOPS_API_MODE true
KHIOPS_DRIVERS_PATH /home/david_trehin_orange_com/atelier_khiops/.venv/lib
Internal environment variables:
None
Temporary file directory: /tmp
System resources:
Host number 1
Physical cores on system 8
Logical processes on system 8
Available memory on system 102.2 GB (Logical 63.9 GB)
Available disk space on system 193.6 GB
hostname MPI ranks logical memory disk cores
vm-ic3-gce 0,1,2,3,4,5,6,7 63.9 GB 193.6 GB 8
System:
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"

[popescu-v]

Assignment for qualification only.

[DTRGit]

Solution proposal :

Bug analysis :
The bug is caused by the date_part column, imported from BigQuery with the dbdate extension type (db_dtypes.DateArray). KhiopsClassifier.fit() cannot serialize this type with pandas 3.0.5, triggering: "AttributeError: 'bool' object has no attribute 'to_numpy'".

Correction patch :
df_Primaire = query_job.to_dataframe()
df_Primaire["date_part"] = pd.to_datetime(df_Primaire["date_part"])

[popescu-v]

Following the qualification, it results that:

• the error lies, indeed, in the handling of the "dbdate" columns: db_dtypes.DateArray is a type supported in the "db-dtypes" package related to BigQuery data, which is not (yet) supported by Khiops (see issue Add BigQuery learning functions #78):
• adding explicit support for "dbdate" in khiops.sklearn.dataset.get_khiops_type seems ad-hoc and brittle in the absence of explicit db_dtypes or pandas-gbq support: if "dbdate" name changes upstream, then there is no traceability in the Khiops Python dependency chain;
• adding support for the two BigQuery-related packages is part of solving issue Add BigQuery learning functions #78 and requires more careful design and work upfront and, for this reason, is not currently on a precise roadmap.

Therefore, a way forward for this issue is to proceed to the following actions:

1. Mitigate the specific bug raised by this issue: as pointed out by @n-voisine, this comes down to restricting the carriage return replacement with white spaces in khiops.skearn.dataset.write_internal_data_table to columns whose (lowercase) NumPy type is stringy (string, str). This mitigates the exception because currently the issues lies in that comparing a "dbdate" array (of type db_dtypes.DateArray) to a scalar results in a single Boolean value rather than an array mask (as it is the case with NumPy arrays or Pandas series or dataframes). Ruling out this comparison for such non-stringy types removes the exception entirely. However, this does not solve the issue completely, because such "dbdate" columns would be determined as being of type "Categorical" by Khiops.
2. Add a Q&A instructing users to convert their "dbdate" columns to Pandas "datetime" before using the Khiops sklearn API.
3. In khiops.sklearn.dataset.get_khiops_type, issue a warning for a third-party Pandas extension type (a Pandas extension type that is not in Pandas itself), before assigning the Khiops "Categorical" type.

An alternative to 3. would be to support explicitly "dbdate" and "dbtime" (convert to Khiops "Date" and "Time" types, respectively), but this is a new feature, seems brittle (because "dbdate" and "dbtime" can change according to upstream package evolutions) and ad-hoc (why only "dbdate" and "dbtime" should be supported, rather than all of BigQuery Python APIs, as provided by google-cloud-bigquery Python package?). IMHO, this support should take place in the PR resolving issue #78 , if roadmapped.