extraction: vendored COBOL grammar never terminates on a bare unterminated token

[ScriptedAlchemy]

Found while proving #1062's end_line equivalence over line-ending shapes: the vendored COBOL tree-sitter grammar (tokensave-large-treesitters, lang-cobol) never terminates when parsing the bare input "x" with no trailing newline. The #1062 shape test therefore targets representative grammars rather than every bundled one.

Reproduce: in crates/tracedecay-code-extraction, parse "x" through the COBOL language (e.g. via ts_provider + CobolExtractor::extract_source, or a scratch test that calls Parser::parse with the COBOL language) — the parse does not return. Establish whether this is a grammar bug (external scanner loop on EOF without newline) in the upstream tree-sitter-cobol revision we vendor, and either bump to a fixed revision or guard at our boundary (the extraction path already has a parse deadline via CorpusParseDeadline; confirm that deadline actually fires for a scanner stuck in a loop — if it does not, that is the real defect, because a hung scanner would pin an indexer worker). Add the COBOL shape to the #1062 equivalence test once it terminates.

[ScriptedAlchemy]

Status: root-caused, fix proven out-of-repo, blocked on an external action (fork of the grammar bundle). Documentation landed at f1cc904 (118c8bd): the input class is recorded at cobol_extractor.rs::parse_source, the deadline's reach at incremental.rs, and the #1062 end_line shape list is gated for COBOL with this issue number.

Root cause. The LINE_PREFIX_COMMENT branch of the vendored tree-sitter-cobol external scanner (tokensave-large-treesitters 0.5.0 → vendor/tree-sitter-cobol/src/scanner.c, upstream c7a36d7) loops while (get_column() <= 5) advance() with no EOF test. Any token starting in 0-based columns 0–5 with no ≥6-column line between it and EOF reaches EOF, where advance is a no-op, and spins forever. Hangs: "x", "x\n", "x\ny\n", "abcde", "é\n界", and a well-formed program followed by a short trailing line such as abc (real-world risk; 17 of 30 probed shapes). Parses: "abcdef", " x", sample.cob with or without trailing newline.

The deadline does not fire. The only parse deadline is incremental.rs::parse_with_deadline_unmeasured (tree-sitter 0.26.11 ParseOptions::progress_callback; set_timeout_micros / set_cancellation_flag no longer exist). parser.c polls the callback only in ts_parser__check_progress, between parse actions — never inside a scanner call. Measured: RetainedParseDocument::open(…, "cobol", "x", max_parse_time = 250 ms) never returned in 5 s. The fresh-parse path (ts_provider::parse_extractor_source_inner) has no deadline at all. So today a COBOL file in this class pins an indexer worker until daemon restart.

Why not a thread guard. The only way to make the deadline fire against a non-cooperative scanner is a detached parse thread abandoned on timeout: ~127 µs per spawn round-trip (roughly doubling a small-file parse) and each hang leaks a 100 %-CPU thread until restart — a corpus like the one in upstream PR #42 (41/41 files hang) would pin 41 cores. Not added.

Upstream. Fix is yutaro-sakamoto/tree-sitter-cobol#42 (open since 2026-07-24, unmerged); the same fix is open in zharinov/tree-sitter-cobol#1 and bearcove/arborium#211; arborium-cobol 2.18.2 still has the bug. No fixed published revision exists. aovestdipaperino/tokensave-large-treesitters has no PR for it and no ScriptedAlchemy fork exists.

Proven patch (compiled against the bundle's own parser.c: all 12 hang-class inputs return in < 1.3 ms, every previously-terminating parse yields a byte-identical S-expression, COBOL passes the #1062 equivalence on all 20 shapes):

--- a/vendor/tree-sitter-cobol/src/scanner.c
+++ b/vendor/tree-sitter-cobol/src/scanner.c
@@ -109,8 +109,18 @@
     if(valid_symbols[LINE_PREFIX_COMMENT] && lexer->get_column(lexer) <= 5) {
-        while(lexer->get_column(lexer) <= 5) {
+        // The column can only advance while input remains: at end of file
+        // `advance` is a no-op, so testing the column alone spins forever
+        // (yutaro-sakamoto/tree-sitter-cobol#42).
+        bool consumed_any = false;
+        while(lexer->lookahead != 0 && lexer->get_column(lexer) <= 5) {
             lexer->advance(lexer, true);
+            consumed_any = true;
+        }
+        // Never emit a zero-width token: the parser would re-enter the scanner
+        // at the same position and loop just as tightly.
+        if(!consumed_any) {
+            return false;
         }
         lexer->result_symbol = LINE_PREFIX_COMMENT;
         lexer->mark_end(lexer);

To land it (owner decision — creates a repo under the org): fork aovestdipaperino/tokensave-large-treesitters at 0fc87352c7 under ScriptedAlchemy, apply the patch, then in the workspace Cargo.toml add [patch.crates-io] tokensave-large-treesitters = { git = "https://github.com/ScriptedAlchemy/tokensave-large-treesitters", rev = "<sha>" } (the same precedent as the tree-sitter-rust#301 pin), update Cargo.lock, add "cobol" to linked_grammar_keys(), and un-gate the COBOL shape in common::tests. Upstream the scanner fix to the bundle as well so the patch can be dropped when it ships.

[ScriptedAlchemy]

Fixed on the #707 branch at cc3042d.

Forked the grammar bundle as ScriptedAlchemy/tokensave-large-treesitters at v0.5.0 (0fc8735) and applied the scanner fix on branch cobol-scanner-eof (102d0103b6fd093c1e5989cceede1ed9f08fe6a9): the LINE_PREFIX_COMMENT skip stops at lookahead == 0 and never emits a zero-width token. The workspace pins it via [patch.crates-io] (same precedent as the tree-sitter-rust pin), cobol joined linked_grammar_keys() in the #1062 end_line shape test, and the interim documentation of the input class was replaced by a note on the pin. The lockfile re-resolve also moved ten Windows-only windows-sys edges from 0.52 to 0.59 (both versions were already locked; no package added).

Evidence on the tip: common::tests with COBOL linked (default features) passes in 0.03 s — "x", "x\n", "x\ny\n", "é\n界" and the other short-trailing-line shapes previously spun forever; --no-default-features --features lang-cobol --test main -- cobol 11/11; crate --all-targets 1,461 tests green; workspace clippy -D warnings clean with --locked.

Upstreamed as aovestdipaperino/tokensave-large-treesitters#6 so the pin can be dropped when a release carries it (the grammar-level fix remains yutaro-sakamoto/tree-sitter-cobol#42, unmerged). The parse deadline's documented reach is unchanged: it is cooperative and cannot interrupt a scanner call, so a future grammar with the same class of bug would need the same treatment.