Read partitioned parquet directories as single tables

[DW10]

Fixes #141

What changed

• datafaker/parquet2orm.py: new find_parquet_tables decides what a table is.
  • A parquet file directly in the input directory is one table (unchanged).
  • A subdirectory with parquet files at any depth is one table named after it.
  • If --parquet-dir has no loose files and only key=value subfolders, it is itself one table.
• Consistency check: all files of a directory table must have the same columns, types and partition keys. Otherwise make-tables logs a clear error naming the differing file and recommending each table be moved into its own directory, then exits 1.
• Hive partition keys (e.g. yr) become TEXT columns with no "Could not determine type" warning.
• datafaker/db_utils.py: each dataset is exposed to DuckDB as a TEMP VIEW named after the directory (read_parquet([...], hive_partitioning=true, union_by_name=true)), so file-backed source databases are not modified. file_search_path now lists only directories with loose files.
• Docs: new "Partitioned Parquet Datasets" section in the parquet tutorial.

Why

Previously every part file became its own table keyed by basename, so identical part names (p00001.parquet) in different tables silently overwrote each other.

Reviewer notes

• Behaviour change: nested folders of plain files (sub/a.parquet, sub/b.parquet) are now one table sub, not two tables.
• A bug fixed along the way: parquet-dir read back from orm.yaml is a string, so it is now converted to Path.
• Partition keys are TEXT in orm.yaml even when DuckDB reads all-numeric keys as integers.
• The schema check opens every file footer; not timed on thousands of files.
• Tests: 11 new in tests/test_parquet2orm.py. Locally on Python 3.14 some existing async tests fail with "no current event loop" on main too; with a test-only event-loop shim the whole suite passes (202). tests/test_rst.py could not run locally.

🤖 Generated with Claude Code

[DW10]

Tested locally on fake data. Able to sift through a mixed format (Some .parquet and some directory with different levels of partition). Also on a single file format or a fixed partition format.

[DW10]

Checked and working on the real data on the GAE server.
Further commit to loosen datatype check constraint as datatype guess may reasonably change between partitions (if one partition is all missing data)

[DW10]

Hi @tim-band and @myyong,

I opened #141 as we have updated our local dataset to use the partition option for Parquet file outputs. This was causing data faker to get confused with the partition files.

Could you please help fix the pre-commit action and review there changes.

Many thanks!
Dylan

[tim-band]

This is great, thanks!
Just one question and one possible docstring update.

[tim-band]

This is great, thanks! Just one question and one possible docstring update.

Actually, one more; Black is failing on github actions. Fix this by running

poetry run pre-commit run -a

And commit whatever it does if you are happy with it.
It's a bit easier if you do this before committing (but after staging the changes you want to commit). Run poetry run pre-commit and it will run faster because it will only consider the staged files.

[DW10]

@tim-band Changes made and pre-commit formatting sorted, for your re-review!

[tim-band]

Thank you Dylan for your PR!