Preserve JATS tables and refresh legacy XML caches

docs/troubleshooting.md

@@ -785,3 +785,51 @@ Run through this checklist when encountering issues:
 - [CLI Reference](reference/cli.md) - Complete command documentation
 - [How to Repair Validation Errors](how-to/repair-validation-errors.md) - Fixing common issues
 - [GitHub Issues](https://github.com/linkml/linkml-reference-validator/issues) - Report bugs
+
+### JATS tables and XML cache refresh
+
+JATS/PMC XML extraction appends pipe-delimited tables after the existing body
+paragraphs. Tables are found throughout the document, including `floats-group`
+when there is no body. Labels and captions form headings; table paragraphs do
+not also appear as body prose. Other existing body paragraphs, such as table
+attribution or alternative descriptions, are retained. Abstract extraction remains the source's job.
+Restricted body notices are checked before any tables are appended.
+
+Each actual table is rendered once, including tables inside nested wrappers.
+Nested tables without their own wrapper use a generic `Nested table` heading.
+Inline text and symbols are retained, block/line breaks become spaces, and
+literal backslashes and pipes in cells are escaped. Rows preserve source cell
+order, including empty cells. A span is printed as `[rowspan=2]` or
+`[colspan=2]` on its source cell: values are not copied into other rows or
+columns. These are quotable source rows, not a reconstructed rectangular grid;
+interpret spanned rows using the original table. Images and non-HTML table
+encodings are not transcribed, but their labels/captions are retained.
+As in existing body extraction, superscript/subscript text is flattened:
+`10<sup>9</sup>` becomes `109`, not exponent notation. Consult the original
+for numeric interpretation; superscript styling is not preserved.
+
+The first **200 source rows per table**, including header and empty rows, are
+kept. Larger tables end with `[Table truncated after 200 rows.]`; later rows
+cannot be validated from this cache. Table footnotes in `table-wrap-foot` are retained once in document order,
+including notes in `floats-group`. The cap is per table, not per document.
+
+`full_text_xml` entries now carry `xml_extraction_version: 1`, independently of
+`extractor_version` and `html_full_text_version`. Missing/older XML stamps cause
+refresh on the next validation fetch; current PDF and HTML entries need no
+refresh for this change. Fresh source/provider XML is stamped after acquisition.
+Inventory and metadata-only rewrites preserve the original XML stamp, including
+future versions, and never certify old text. If refresh is unavailable, legacy
+XML remains available with the existing stale-cache warning and is not rewritten;
+it may still lack table rows. A later process retries the refresh. Existing
+stale HTML rejection remains unchanged.
+
+A successful source refresh that returns only an abstract can replace the old
+full text if no provider supplies a body. This is existing refresh behavior;
+the stale fallback applies when the source returns no record, not when it
+returns an abstract-only record. Keep a backup if retaining older full text is
+necessary.
+
+This pass targets JATS `table-wrap` content and searches the whole document;
+tables and notes inside embedded `sub-article` or `response` elements are
+excluded so reviewer/reply findings are not attributed to the main paper. Bare tables
+without a `table-wrap` remain outside this JATS extraction pass.

src/linkml_reference_validator/etl/extract/__init__.py

@@ -5,7 +5,10 @@
 When a change means previously cached text is *wrong* rather than merely older,
 bump ``EXTRACTOR_CACHE_VERSION`` in
 :mod:`linkml_reference_validator.etl.reference_fetcher` so existing entries are
-re-fetched instead of keeping the old output forever.
+re-fetched instead of keeping the old output forever. For changes confined to
+XML table extraction, bump ``XML_EXTRACTION_CACHE_VERSION`` instead; HTML
+acceptance has its own ``HTML_FULL_TEXT_CACHE_VERSION``. These format-specific
+stamps avoid refreshing unchanged formats.
 """
 
 from linkml_reference_validator.etl.extract.base import Extractor, ExtractorRegistry

src/linkml_reference_validator/etl/extract/xml.py

@@ -2,13 +2,14 @@
 
 Changing what this yields for the same input may make already-cached text wrong
 rather than merely older; see :mod:`linkml_reference_validator.etl.extract` for
-when to bump ``EXTRACTOR_CACHE_VERSION``.
+when to bump cache versions. Table changes use ``XML_EXTRACTION_CACHE_VERSION``
+so PDF and HTML entries remain current.
 """
 
 import logging
 from typing import Optional, Union
 
-from bs4 import BeautifulSoup  # type: ignore
+from bs4 import BeautifulSoup, CData, NavigableString, Tag  # type: ignore
 
 from linkml_reference_validator.etl.extract.base import Extractor, ExtractorRegistry
 
@@ -83,18 +84,117 @@ def is_stub_notice(text: str) -> bool:
     return any(phrase in lowered for phrase in STUB_NOTICE_PHRASES)
 
 
+#: Bound cached table evidence; headers count and truncation is always explicit.
+MAX_TABLE_ROWS = 200
+
+
+def _table_text(node: Tag) -> str:
+    """Keep inline text contiguous, separating blocks and excluding nested tables."""
+
+    def walk(tag: Tag) -> str:
+        """Render descendants without mutating the parsed document."""
+        parts = []
+        for child in tag.children:
+            if type(child) in (NavigableString, CData):
+                parts.append(str(child))
+            elif isinstance(child, Tag) and child.name not in {"table", "table-wrap"}:
+                value = walk(child)
+                if child.name in {"p", "break", "br", "list-item", "title"}:
+                    value = " " + value + " "
+                parts.append(value)
+        return "".join(parts)
+
+    return " ".join(walk(node).split())
+
+
+def _table_heading(wrap: Tag) -> str:
+    """Read only the label/caption owned by this wrapper, not nested wrappers."""
+    parts = []
+    for name in ("label", "caption"):
+        node = next(
+            (n for n in wrap.find_all(name) if n.find_parent("table-wrap") is wrap),
+            None,
+        )
+        if node is not None:
+            parts.append(_table_text(node))
+    return " ".join(part for part in parts if part)
+
+
+def _tables_as_text(soup: BeautifulSoup) -> list[str]:
+    """Render each table once in document order, using source cells, not a grid.
+
+    Row/column spans are explicit annotations; values are never replicated or
+    assigned to inferred columns. Header rows count toward the 200-row limit.
+    """
+    sections = []
+    for table in soup.find_all(["table-wrap", "table", "table-wrap-foot"]):
+        if table.find_parent(["sub-article", "response"]):
+            continue
+        if table.name == "table-wrap":
+            heading = _table_heading(table)
+            if heading:
+                sections.append("## " + heading)
+            continue
+        wrap = table.find_parent("table-wrap")
+        if wrap is None:
+            # This pass targets JATS tables, not arbitrary XML layout tables.
+            continue
+        if table.name == "table-wrap-foot":
+            note = _table_text(table)
+            if note:
+                sections.append(note)
+            continue
+        # Named wrapper headings were emitted at their own document position.
+        heading = "" if _table_heading(wrap) else "Table"
+        if table.find_parent(["table", "table-wrap"]) is not wrap:
+            heading = "Nested table"
+        rows = [
+            row for row in table.find_all("tr") if row.find_parent("table") is table
+        ]
+        rendered = []
+        for row in rows[:MAX_TABLE_ROWS]:
+            cells = []
+            for cell in row.find_all(["th", "td"]):
+                if (
+                    cell.find_parent("tr") is not row
+                    or cell.find_parent("table") is not table
+                ):
+                    continue
+                value = _table_text(cell).replace("\\", "\\\\").replace("|", "\\|")
+                for span in ("rowspan", "colspan"):
+                    if cell.has_attr(span) and str(cell[span]) != "1":
+                        value += f" [{span}={cell[span]}]"
+                cells.append(value)
+            if cells:
+                rendered.append("| " + " | ".join(cells) + " |")
+        if len(rows) > MAX_TABLE_ROWS:
+            rendered.append(f"[Table truncated after {MAX_TABLE_ROWS} rows.]")
+        if rendered:
+            sections.append(
+                ("## " + heading + "\n\n" if heading else "") + "\n".join(rendered)
+            )
+    return sections
+
+
 @ExtractorRegistry.register
 class XMLExtractor(Extractor):
     """Extract body text from JATS/PMC article XML.
 
-    Returns the concatenated text of paragraphs within the article ``<body>``.
-    Returns None when there is no body content, and when the body holds one of
+    Returns main-article body paragraphs followed by labeled tables and notes,
+    including floats-group content. Sub-article and response content is excluded.
+    Table cells retain inline text and explicitly annotate spans; the first 200
+    source rows per table are retained, with a notice when rows are omitted.
+    Returns None when neither body paragraphs nor table content exists, and when
+    the body holds one of
     PMC's placeholder notices instead of the article itself.
 
     Examples:
         >>> xml = b"<article><body><p>Hello body.</p></body></article>"
         >>> XMLExtractor().extract(xml)
         'Hello body.'
+        >>> table = "<article><table-wrap><table><tr><td>Yes</td></tr></table></table-wrap></article>"
+        >>> XMLExtractor().extract(table)
+        '## Table\\n\\n| Yes |'
         >>> stub = b"<article><body><p>Text cannot be obtained from PMC.</p></body></article>"
         >>> XMLExtractor().extract(stub) is None
         True
@@ -112,17 +212,28 @@ def extract(
         # re-encoding str here would leave that declaration contradicting the
         # bytes. The parser gets both cases right on its own.
         soup = BeautifulSoup(data, "xml")
-        body = soup.find("body")
-        if not body:
-            return None
-
-        paragraphs = body.find_all("p")
-        if not paragraphs:
-            return None
-
-        text = "\n\n".join(p.get_text() for p in paragraphs if p.get_text().strip())
-        if not text.strip():
-            return None
+        body = next(
+            (
+                node
+                for node in soup.find_all("body")
+                if not node.find_parent(["sub-article", "response"])
+            ),
+            None,
+        )
+        paragraphs = (
+            [
+                p
+                for p in body.find_all("p")
+                if not p.find_parent(["sub-article", "response"])
+            ]
+            if body
+            else []
+        )
+        text = "\n\n".join(
+            p.get_text()
+            for p in paragraphs
+            if not p.find_parent("table-wrap") and p.get_text().strip()
+        )
 
         # Judged on the extracted body, not the raw markup: a citation title or
         # a methods sentence elsewhere in the document says nothing about
@@ -137,4 +248,17 @@ def extract(
             )
             return None
 
-        return text
+        # Keep existing attribution/alternative prose that the renderer does not
+        # emit. Table content must not inflate the stub check above.
+        text = "\n\n".join(
+            p.get_text()
+            for p in paragraphs
+            if p.get_text().strip()
+            and not (
+                p.find_parent("table-wrap")
+                and p.find_parent(["label", "caption", "table", "table-wrap-foot"])
+            )
+        )
+        return (
+            "\n\n".join(part for part in [text, *_tables_as_text(soup)] if part) or None
+        )

src/linkml_reference_validator/etl/reference_fetcher.py

@@ -64,6 +64,9 @@
 #: PMC or a configured text provider is trusted under FullTextLocation's contract.
 HTML_FULL_TEXT_CACHE_VERSION = 1
 
+#: XML table extraction changes only XML caches, independent of HTML acceptance.
+XML_EXTRACTION_CACHE_VERSION = 1
+
 #: A cache file's frontmatter delimiter: a line that is exactly ``---``.
 #: Splitting on the bare string instead lets any *value* containing ``---`` - a
 #: URL reference_id, a title - truncate the block, which loses every field after
@@ -184,6 +187,11 @@ def fetch(
                 content.metadata or {}, html_full_text_version=HTML_FULL_TEXT_CACHE_VERSION
             )
 
+        if content and content.content_type == "full_text_xml":
+            content.metadata = dict(
+                content.metadata or {}, xml_extraction_version=XML_EXTRACTION_CACHE_VERSION
+            )
+
         if content and self.config.fetch_full_text and self.needs_full_text(content):
             content = self._enrich_with_full_text(content)
 
@@ -403,6 +411,10 @@ def _apply_full_text_location(
             content.metadata = dict(
                 content.metadata or {}, html_full_text_version=HTML_FULL_TEXT_CACHE_VERSION
             )
+        if fmt == "xml":
+            content.metadata = dict(
+                content.metadata or {}, xml_extraction_version=XML_EXTRACTION_CACHE_VERSION
+            )
         content.full_text_provider = location.provider or provider_name
         # Non-public endpoints are not durable provenance and may contain
         # session-specific access information.
@@ -711,6 +723,10 @@ def _save_to_disk(
         html_version = (reference.metadata or {}).get("html_full_text_version")
         if reference.content_type == "full_text_html" and isinstance(html_version, int):
             lines.append(f"html_full_text_version: {html_version}")
+        # YAML booleans are not extraction versions, despite bool subclassing int.
+        xml_version = (reference.metadata or {}).get("xml_extraction_version")
+        if reference.content_type == "full_text_xml" and type(xml_version) is int:
+            lines.append(f"xml_extraction_version: {xml_version}")
         if reference.title:
             lines.append(f"title: {self._quote_yaml_value(reference.title)}")
         if reference.authors:
@@ -976,7 +992,7 @@ def _split_frontmatter(content_text: str) -> Optional[tuple[str, str]]:
 
     @classmethod
     def _is_stale_cache_entry(cls, content_text: str) -> bool:
-        """Report whether extraction or HTML full-text acceptance needs refreshing.
+        """Report whether extraction or format-specific full-text processing needs refreshing.
 
         Deliberately not applied by :meth:`iter_cached_references`: export and
         enrichment walk the cache as a record of what was fetched, and dropping
@@ -1017,6 +1033,11 @@ def _is_stale_cache_entry(cls, content_text: str) -> bool:
             ):
                 return True
 
+        if isinstance(metadata, dict) and metadata.get("content_type") == "full_text_xml":
+            xml_version = metadata.get("xml_extraction_version")
+            if type(xml_version) is not int or xml_version < XML_EXTRACTION_CACHE_VERSION:
+                return True
+
         # A newer stamp is not stale: an older tool reading a cache written by a
         # newer one should leave it alone rather than re-fetch it on every run.
         return False
@@ -1056,6 +1077,8 @@ def _load_markdown_format(
         )
 
         metadata: dict = {}
+        if "xml_extraction_version" in frontmatter:
+            metadata["xml_extraction_version"] = frontmatter["xml_extraction_version"]
         if "html_full_text_version" in frontmatter:
             metadata["html_full_text_version"] = frontmatter["html_full_text_version"]
         if "extra_fields_captured" in frontmatter: