[Refactor] DoclingDocument as the OCR layout contract

.gitignore

@@ -165,6 +165,8 @@ extralit/site
 # Development files
 **/*.db
 **/*.pdf
+# Layout parser test fixtures are committed; regenerate with extralit-server/tests/fixtures/pdf/generate.py
+!extralit-server/tests/fixtures/pdf/*.pdf
 output/
 .output/
 .nuxtrc

docs/admin_guide/pdf_preprocessing_config.md

@@ -0,0 +1,120 @@
+# PDF Preprocessing Configuration Guide
+
+Extralit Server runs [OCRmyPDF](https://github.com/ocrmypdf/ocrmypdf) over every uploaded PDF for
+**page rotation only**. No OCR text is produced: `tesseract_timeout=0` kills the tesseract OCR
+spawn and `skip_text` leaves pages that already have text untouched. The one thing tesseract is
+still asked for is OSD — deciding whether a page is sideways — which is why its budget is bounded.
+
+Settings live in `PDFPreprocessingSettings` and are configurable via `PREPROCESSING_`-prefixed
+environment variables.
+
+## What runs on an upload
+
+| Step | Component | Output |
+|---|---|---|
+| Triage | `contexts/ocr/triage.py` (pdf-inspector) | `pdf_type`, `pages_needing_ocr`, `pages_with_tables`, `pages_with_columns`, encoding issues |
+| Margins + thumbnail | `contexts/document/margin.py` (`PDFAnalyzer`) | `analysis_metadata.layout_analysis.margin_analysis`, thumbnail object |
+| Rotation | `contexts/document/preprocessing.py` (ocrmypdf) | the PDF rewritten at the same key |
+| Layout | `jobs/ocr_jobs.py` | canonical `DoclingDocument` JSON + the workspace's Lance rows |
+
+Triage classifies which pages have no usable text, but pdf-inspector bundles no OCR engine and
+tesseract OCR is off, so those pages are **surfaced, not fixed**: they appear in
+`analysis_metadata.triage.pages_needing_ocr` and `layout_metadata.pages_needing_ocr` and stay an
+explicit gap until an OCR job exists.
+
+## Configuration Reference
+
+### `PREPROCESSING_ENABLED`
+- **Type**: `bool` — **Default**: `true`
+- Master switch. When `false`, the PDF is passed through byte-identical; triage, margins and the
+  thumbnail still run (they are the analysis job's own work, not the preprocessor's).
+
+### `PREPROCESSING_ROTATE_PAGES`
+- **Type**: `bool` — **Default**: `true`
+- Auto-rotate pages whose text is not upright.
+
+### `PREPROCESSING_ROTATE_PAGES_THRESHOLD`
+- **Type**: `float` — **Default**: `2.0`
+- Confidence OSD must reach before a page is rotated. Lower (1.0–1.5) rotates more eagerly;
+  higher (3.0+) avoids false rotations.
+
+### `PREPROCESSING_TESSERACT_NON_OCR_TIMEOUT`
+- **Type**: `float` (seconds per page) — **Default**: `30.0`
+- Budget for OSD, the only tesseract call made here. OCRmyPDF's own default is 180 s per page,
+  which dominates the runtime on image-heavy PDFs.
+
+### `PREPROCESSING_JOBS`
+- **Type**: `int` — **Default**: `1`
+- Worker processes for ocrmypdf. Keep at `1` in containers with limited CPU to avoid
+  oversubscription; `2–4` on a multi-core host.
+
+### `PREPROCESSING_PROGRESS_BAR`
+- **Type**: `bool` — **Default**: `false`
+- Useful interactively, noise in background jobs.
+
+### Fixed, not configurable
+
+`skip_text=True`, `tesseract_timeout=0`, `clean=False`, `optimize=0`. These are what make the pass
+rotation-only: `clean` (unpaper) and `optimize` only pay off alongside OCR output, and the
+alternatives that would OCR image pages are destructive — `force_ocr` rasterizes the existing text
+layer, and `redo_ocr` strips invisible OCR text it cannot regenerate with tesseract disabled.
+
+### Known limit
+
+Verified in `ocrmypdf/_pipeline.py::is_ocr_required`: under `skip_text`, OSD only runs on pages
+ocrmypdf would process, i.e. image-only pages. A born-digital page keeps whatever `/Rotate` it
+already has. A text-only PDF still pays pdfinfo, a re-save and the S3 rewrite — cheap, and nothing
+is rasterized.
+
+## Troubleshooting
+
+### Rotation is slow on scanned PDFs
+
+OSD is the cost. Lower `PREPROCESSING_TESSERACT_NON_OCR_TIMEOUT`, or set
+`PREPROCESSING_ROTATE_PAGES=false` to skip orientation detection entirely.
+
+### A page is rotated the wrong way
+
+Raise `PREPROCESSING_ROTATE_PAGES_THRESHOLD` so OSD needs more confidence before acting.
+
+### Rotation failed
+
+The job records it and keeps going: the original bytes are stored, and
+`preprocessing_metadata.rotation_ran` is `false` with the reason in `preprocessing_metadata.error`.
+Nothing downstream is blocked, because layout and text extraction depend on this job with
+`allow_failure`.
+
+### High memory usage
+
+Set `PREPROCESSING_JOBS=1`. Rotation itself holds one page image at a time.
+
+## Integration Example
+
+```python
+from extralit_server.contexts.document.preprocessing import (
+    PDFPreprocessingSettings,
+    PDFPreprocessor,
+)
+
+settings = PDFPreprocessingSettings(rotate_pages=True, tesseract_non_ocr_timeout=15.0, jobs=2)
+result = PDFPreprocessor(settings).preprocess(pdf_bytes, "document.pdf")
+
+print(result.metadata.rotation_ran, result.metadata.processing_time, result.metadata.error)
+processed_pdf = result.processed_data
+```
+
+## Related Components
+
+| File | Purpose |
+|------|---------|
+| [`preprocessing.py`](../../extralit-server/src/extralit_server/contexts/document/preprocessing.py) | The rotation pass and its settings |
+| [`triage.py`](../../extralit-server/src/extralit_server/contexts/ocr/triage.py) | Structural classification (pdf-inspector) |
+| [`margin.py`](../../extralit-server/src/extralit_server/contexts/document/margin.py) | Margin detection and thumbnail, over the leading pages |
+| [`document/metadata.py`](../../extralit-server/src/extralit_server/api/schemas/v1/document/metadata.py) | What lands in `documents.metadata_` |
+
+## Further Reading
+
+- [OCRmyPDF Documentation](https://ocrmypdf.readthedocs.io/)
+- [Extralit](https://github.com/Extralit/extralit)
+- [Extralit HF Space](https://github.com/Extralit/extralit-hf-space)
+- [Papers OCR Benchmarks](https://github.com/Extralit/papers-ocr-benchmarks)

extralit-frontend/v1/domain/entities/document/DocumentLayout.ts

@@ -0,0 +1,136 @@
+/**
+ * Extracted document layout, mirroring the server's `DoclingDocument` projection.
+ *
+ * Every bounding box is in page points with a top-left origin, relative to the `LayoutPage`
+ * of the same `pageNo` — a viewer must scale by its own rendered page size, never assume 72dpi.
+ */
+
+export interface Rect {
+  left: number;
+  top: number;
+  width: number;
+  height: number;
+}
+
+export class BoundingBox {
+  constructor(
+    public readonly l: number,
+    public readonly t: number,
+    public readonly r: number,
+    public readonly b: number,
+    public readonly coordOrigin: string = "TOPLEFT"
+  ) {}
+
+  get width(): number {
+    return this.r - this.l;
+  }
+
+  get height(): number {
+    return this.b - this.t;
+  }
+
+  /** Scale into the coordinate space of a rendered page of the given size. */
+  toRect(pageWidth: number, pageHeight: number, renderedWidth?: number, renderedHeight?: number): Rect {
+    const scaleX = (renderedWidth ?? pageWidth) / pageWidth;
+    const scaleY = (renderedHeight ?? pageHeight) / pageHeight;
+
+    return {
+      left: this.l * scaleX,
+      top: this.t * scaleY,
+      width: this.width * scaleX,
+      height: this.height * scaleY,
+    };
+  }
+
+  /** Fractions of the page, for overlays that position with percentages. */
+  toRelativeRect(pageWidth: number, pageHeight: number): Rect {
+    return {
+      left: this.l / pageWidth,
+      top: this.t / pageHeight,
+      width: this.width / pageWidth,
+      height: this.height / pageHeight,
+    };
+  }
+}
+
+export class Provenance {
+  constructor(
+    public readonly pageNo: number,
+    public readonly bbox: BoundingBox,
+    /** Item-local character span — an offset into this item's own text, not the document. */
+    public readonly charspan: [number, number]
+  ) {}
+}
+
+export class LayoutItem {
+  constructor(
+    /** Citation anchor, e.g. `#/texts/12`. Stable for the lifetime of the stored layout. */
+    public readonly selfRef: string,
+    public readonly label: string,
+    public readonly readingOrder: number,
+    public readonly prov: Provenance[] = [],
+    public readonly parentRef: string | null = null,
+    public readonly contentLayer: string | null = null,
+    public readonly level: number | null = null,
+    public readonly text: string | null = null,
+    public readonly html: string | null = null
+  ) {}
+
+  /** Every page this item touches; more than one when it spans a page break. */
+  get pageNumbers(): number[] {
+    return [...new Set(this.prov.map((p) => p.pageNo))].sort((a, b) => a - b);
+  }
+
+  get isTable(): boolean {
+    return this.label === "table";
+  }
+
+  get isPicture(): boolean {
+    return this.label === "picture";
+  }
+
+  get isHeading(): boolean {
+    return this.label === "section_header" || this.label === "title";
+  }
+
+  provenanceOnPage(pageNo: number): Provenance[] {
+    return this.prov.filter((p) => p.pageNo === pageNo);
+  }
+}
+
+export class LayoutPage {
+  constructor(
+    public readonly pageNo: number,
+    public readonly width: number,
+    public readonly height: number
+  ) {}
+}
+
+export class DocumentLayout {
+  constructor(
+    public readonly documentId: string,
+    public readonly doclingVersion: string,
+    public readonly pages: LayoutPage[] = [],
+    public readonly items: LayoutItem[] = []
+  ) {}
+
+  get numPages(): number {
+    return this.pages.length;
+  }
+
+  get numItems(): number {
+    return this.items.length;
+  }
+
+  page(pageNo: number): LayoutPage | undefined {
+    return this.pages.find((p) => p.pageNo === pageNo);
+  }
+
+  itemsOnPage(pageNo: number): LayoutItem[] {
+    return this.items.filter((item) => item.pageNumbers.includes(pageNo));
+  }
+
+  itemByRef(selfRef: string): LayoutItem | undefined {
+    return this.items.find((item) => item.selfRef === selfRef);
+  }
+}