Repository navigation
Conversation
New folds in coverage.frm (cov_stmt_*, cov_pp_*, cov_opt_format) exercise statements that were never compiled/run by the test suite: Fill/PrintTable/ ClearTable, Compress, FunPowers, UnitTrace, Ratio, Chisholm, Switch/Case/Break/Default/EndSwitch, If/ElseIf/Else/EndIf with dollar comparisons, SetExitFlag, Exit, Write/NWrite options, ThreadBucketSize, Format optimization options (O1-O4 variants), #optimize/#clearoptimize/ #skipextrasymbols/Delete extrasymbols, #setrandom + random_, #breakdo, #show, #debug, #commentchar, #append/#write/#close and #exchange for expressions and $variables.
…ace options New folds (cov_printoptions, cov_onoff, cov_onoff_error, cov_factdollar, cov_factdollar_error, cov_modulus, cov_traceopts, cov_traceopts_error) cover On/Off statement option keywords and arguments (flag n, compress gzip|zstd level, indentspace n, checkpoint intervals), Print/NPrint with +s/+ss/-s/-f options and the "%$"-string form, the FactDollar statement with factored-dollar indexing in If and Do, the Modulus statement with its full option set, and Trace4/TraceN with option keywords, plus error paths for these statements.
…printing New folds (cov_createall, cov_replaceloop, cov_mergestuffle, cov_sumstmt, cov_putinside, cov_factorize, cov_symmetrize_stmt, cov_chisholm_dollar, cov_printfiles, cov_checkpoint_on, cov_external, cov_createrrors) cover CreateAll loops/paths, ReplaceLoop and the findloop() If condition, Shuffle/Merge/Stuffle with their once variants, the Sum statement, PutInside after Bracket, Factorize/UnFactorize, Symmetrize/AntiSymmetrize/ CycleSymmetrize/RCycleSymmetrize, Chisholm with dollar level and symmetrize options, Print to log-only and to open channels, On checkpoint with run scripts, the #external/#rmexternal preprocessor block and error paths of CreateAll.
…matching New folds (cov_preifdollar, cov_prewrite, cov_symmetricmatch, cov_factdollar2, cov_replaceloop2) cover $(...) dollar conditions of #if with #elseif/#else/#ifdef/#ifndef, the #create/#write/#append/#close/ #remove channel instructions (including the <file> form of #write), identification of symmetric/cyclic/reversecyclic functions, FactDollar on a multi-term dollar, and ReplaceLoop without include and with all loop sizes.
New folds (cov_writeformats, cov_tryreplace, cov_table_err) and a rewritten cov_gettable_err cover the %s/%d/%E/%e/%X/%O format specifiers of #write, the empty-buffer writes to channels, TryReplace with swapped object pairs, sparse CTable lookups with unfilled elements, bounded-range Tables and GetTable error paths.
…PolyFunExp code as excluded from coverage Comment-only change: the Save translation branch that only a 32-bit build can take, and the unfinished PolyFunExp experiments (their entry point is never called), get lcov exclusion markers. Coverage numbers are otherwise pulled down by code that cannot be tested by design.
…o coverage.frm sources: exclude unreachable save file header translation in ReadSaveIndex The transFlag body of ReadSaveIndex is only executed when reading a save file that was written with a different word size or byte order. Following the existing exclusion of the write-side translation code, mark it with UNFINISHED_FEATURE_EXCL markers.
…ode as excluded from coverage Comment-only change in grcc.cc: "cannot happen" internal-error guards and debug-only print routines that no test input can execute. Marked with the same lcov exclusion markers used elsewhere in sources/.
…tors as excluded from coverage Comment-only change: helper functions in ratio.c that are never called, plus grcc.cc constructors that no statement input can reach, get lcov exclusion markers so they stop counting against measured test coverage.
Adds tests covering statement corner cases (Fill, AntiFunction, Symmetrize, ToInteger, argument lists) and error paths (illegal table indices, bad Modulus arguments, malformed preprocessor dollar references). Five tests that use dollar variables cannot run under the threaded binary, so they are marked pending there instead of being removed.
…using garbage compiler state
Adds tests for: factorization statements (Factor, Factorize, FactDollar with powers), preprocessor #if conditions on expressions, Optimise combinations, pattern matching against functions, set membership errors, table filling and Do-loop corner cases, and ExtraSymbols handling. Most tests assert a specific compiler/preprocessor diagnostic; the rest run clean paths that no existing test reached.
…paced expressions Covers: gamma traces with and without contractions (epsilon and dimension symbols), expressions with very large exponents or repeated big coefficients that exercise the big-term storage paths, expressions declared inside namespaces, Keep-with-TermsInBracket behaviour, DoFortran output of a bracket's contents, and error paths for improper Set arguments and missing modulus inverses.
…setup directives Covers: epsilon and large gamma traces, MakeInteger with huge exponents, sparse-table deallocation and negative indices, Clear on dictionaries, SetElement with wrong object types, bracket functions with repeated classes, namespaced expression reuse, and the #setup preprocessor directives (include directory, basic setup). Mix of clean runs and exact diagnostics.
…p options Covers: factored dollars with commas and number factors, swapped powers in dollar expressions, negative minimum-vector counts, MakeRational with a subexpression, trace4 with repeated dimension indices, SetEquality syntax errors, the #setup include-directory option with a comma, and zero-argument statement errors.
…tions Each test triggers (or cleanly exercises) an exact diagnostic or option branch that no existing test reached: malformed ranges and permutations, SetEquality syntax errors, bad quoted extensions in process statements, EndSort verbosity, #write into compressed (gzip) pipes, checkpoint warnings, namespace function declarations, mixed-element brackets, ToTensor on non-vectors, and the deprecation path of "On checkpoint".
…s and pattern subsets Covers: table memory deallocation paths, the Optimisation character options of Factorize, Bernoulli-function pair handling, gamma pattern subsets of dotted products, IsDefined and isFactorized argument checks, set-membership conditions (setelement with symbols and numbers, repeated SetSkip), long string arguments, antisymmetric pattern replacement, and UnFactor for all functions.
A finer re-scan of an earlier batch of generated test inputs found 67 that exercise previously unexecuted lines (detected with finer granularity than the first pass, so multi-line error messages that overlap were no longer treated as duplicates). Adds: table-base creation/reuse errors, model and particle declarations with wrong arguments, factored argument with large denominators, tensor set-code with gamma indices, encode of big numbers, dictionary open errors, match with index substitution, modulus without a coprime inverse, nested namespaces, unterminated quotes/brackets, and similar diagnostics. Each test asserts its exact message.
…from coverage Comment-only change: 65 spans (grcc.cc, mytime.cc) that a coverage audit proved cannot be executed by any test input - unfinished-feature stubs, debug-only printers, and internal-error guards. Marked with the sanctioned lcov exclusion markers (UNFINISHED_FEATURE_EXCL, DEBUG_EXCL, INTERNAL_ERROR_EXCL), one-line justification per span, no live line inside any span. Excludes 798 zero-hit lines from the measurement instead of chasing untestable code with contrived tests.
Exercises the option branches of the graph/diagram statement pair
("Diagram"/"GenerateDiagram" style declarations with TopologiesOnly,
WithEdges, WithBlocks, OnePISets, Symmetrize, OnShell combinations) and
their error paths: odd-parity vacuum diagrams, too many couplings, missing
vertices, particle overflow, wrong external-momentum counts for
tadpoles/snails/two-point functions, fermion-line splitting for
Majorana/Dirac/ghost lines, and non-simple phi4 cases. Option names and
behaviour taken from the manual's diagrams documentation.
…r messages Each test is a tiny program that triggers exactly one diagnostic: invalid expression and particle names, bad keywords, missing or extra brackets, illegal indices and powers, bad tensor/symmetry/table declarations, misuse of namespace builtins, malformed preprocessor variables, and similar input mistakes. Multi-error tests cannot share a file (the test driver stops a file's execution at the first failing program), so every error case is its own fold. Eight multi-program folds cover clean corner paths (Optimize without expressions, big scratch writes, count of large functions, factorized argument extensions).
…nd dollar corner cases Covers: TableBase open/addto/destroy error paths and their clean paths, Compress with gzip at program end, ProcessBucket with wrong numbers, the SortSum option of ProcessBucket, canonical ordering of topologies, ReplaceBody with subexpression wildcards, Encode/Decode of large numbers, implicit multiplication in expressions, dollar variables in function indices and in loops, #setup define/env-var handling, #write with include files, and trace4 with dollars. Clean cases share multi-program folds; error cases stay individual.
…rs) as excluded from coverage Comment-only change, following the standard lcov exclusion markers: TestTerm has zero callers in the whole source tree, MakeDollarMod's only caller is itself unused, and several grcc.cc branches are proven unreachable (unused "defined by name" variants, internal-error guards that cannot fire because parse rejects such inputs earlier). 214 zero-hit lines stop counting against measured coverage; no executable line is touched.
…or the wrong reason A test audit found 5 tests asserting incorrect FORM behaviour as their expected result (Stuffle of a root-of-unity symbol adds instead of multiplying, MakeRational and Modulus hitting internal errors but the test still passing on the printed value). Those are replaced by tests of the correct semantics where one exists, or removed where fixing them is impossible without fixing the underlying bug (Stuffle case removed and the bug documented). About 30 error tests tripped an earlier, unintended error because of invalid syntax, so their assertions passed while the intended statement was never executed: syntax corrected so each intended diagnostic is actually reached. Multi-program tests whose later programs never ran (the driver stops at the first failing program) are split up.
…ions Upstream CI sets FORM_IGNORE_DEPRECATION=1 for every job and also runs a 32-bit (i386) job: three checkpoint tests asserted that the checkpoint deprecation warning is printed, which fails when that variable is set, and one test byte-patches a 64-bit store file with python3, which cannot work in the 32-bit job. The checkpoint tests now accept a run either with or without the warning (the functional asserts remain), the store-file test is dropped until it can be made word-size aware, and include files are written to the per-test temporary directory instead of the test-source directory. All suites verified green with the environment variable both set and unset.
The two MesPrint calls in CoTBon/CoTBoff contained a %s conversion without the corresponding argument. MesPrint then reads an unrelated value off the varargs stack and dereferences it: on x86-64 gcc this happened to print garbage after the name that was still live in a register, but under clang (macOS runners) it crashed FORM with a segmentation fault in MesPrint instead of the intended error message. Reached in the test suite for the first time by the new "TableBase On with an undefined subkey" coverage test; pass `s`, which is the NUL-terminated subkey at that point.
The ParFORM job of the test suite runs each test under mpirun -np 4, where three of the new coverage test groups fail: - 30 tests hang to the 15 s timeout: when the master hits a runtime error (SetExitFlag) or a compile/preprocessor error, the workers are never told about the termination and mpirun waits forever, so these tests cannot work under ParFORM until its termination protocol changes. - 23 dollar-variable tests are forced into sequential mode (the warning already trips the same check under tform) and needed the same mpi? extension that threaded-only tests already carry. - 9 tests pin output that differs under parallel execution (random_ seed paths, argument-sort order, range-error semantics and two folds that are simply slower than the 15 s timeout). All 62 now carry '#pend_if mpi?' (or '|| mpi?' appended where a threaded? guard already exists), mirroring the check for the existing folds that skip under ParFORM. The failure list was established by reproducing the exact CI parform job (OpenMPI, timeout 15) locally and re-checking every failing fold. On/Off flint is also split out of the option-sweep test into its own cov_onflint fold: on FLINT-less builds it only emits a warning, so it now asserts 'succeeded? || warning?' like the existing polynomial tests, instead of failing the 32-bit and flint-less CI jobs. Verified: vorm/tvorm default suites green with and without FORM_IGNORE_DEPRECATION, the FLINT-less builds green, the extra suite green, and the 62 folds pended under the reproduced ParFORM job.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR raises the line coverage of the FORM test suite (gcov/LCOV, default suites, form+tform merged) from 62.0% to 81.97%, and function coverage from 76.6% to 94.5%, using ~1,140 new tests in
check/coverage.frm.sources/filesNo existing test changed outcome. Upstream CI-relevant configurations were verified by execution: both x86-64 binaries pass with
FORM_IGNORE_DEPRECATIONunset and set to 1 (the upstream CI sets it globally), theextrasuite passes (1343 tests), and no test depends on non-portable behaviour (one test that byte-patched a store file could not pass the 32-bit CI job and was removed rather than shipped broken).Almost all tests live in
check/coverage.frm, which already exists upstream as the designated home for coverage-driven tests. Fold names are prefixedcov_and are unique across the wholecheck/directory; the standard*--#[ ... *--#]fold format is used throughout.How the tests were produced
Writing the tests was an iterative, evidence-driven loop — enumerate uncovered spans of code, generate probe programs that try to reach them, and keep only the probes whose effect is verified by execution. The basic idea follows the strategy highlighted in 2609.23367. The target in this case is the source code, instead of the documentation. FORM-8B is used in a repair loop to fix the syntax, lcov is the oracle.
The main strategies, in the order they were stacked:
assert compile_error?/preprocess_error?/runtime_error?folds following the existing precedents incheck/.Validation
fastandtvormsuites, and theextrasuite): 1699/1699 tests, 0 failures, in both deprecation-env variants.glm-5.3anddeepseek-v4-flash), not just any output (this caught, e.g., a Stuffle test that happily asserted a wrong coefficient).Adversarial review
The audit rounds deliberately tried to fail the test suite, and they did:
MakeRational/Modulustests that passed on values printed after an internal error). These were replaced with tests of the correct semantics where one exists, or removed (the suite must not validate bugs!).check/), removal of an i386-incompatible test, and checkpoint-warning assertions rewritten to hold in both CI environment variants.Tentative bug count
Running ~1,100 new inputs through FORM exposed a sizeable bug surface, which were documented separately (triaged reports will follow as individual issues once validated more carefully; none of the still-open bugs are asserted as "expected" by any test):
sources/commits in this branch):ifmatch/Transformthat fail pattern compilation left uninitialized garbage to be executed downstream; they now bail out cleanly (44 of 63 surveyed crash inputs traced back to this pair alone).#matchwithf(a?$w)under$wreuse).warn + continuebehaviour, retracted).The tentative headline: 2 fixed, ~23 report-ready, several hundred inputs collected for triage.
Remaining uncovered tail is long and scattered
From ~82%, every additional point gets progressively more expensive, and the tail decomposes into four quantified categories:
else-branches, thread-scheduler internals) would remove another several thousand lines from the denominator. Because exclusion shrinks the denominator, further flagging moves the percentage almost as cheaply as tests do.Modulus/MakeRationalinternals) are entered only by inputs that currently crash or abort, so they cannot be covered by tests until the underlying bugs are fixed: fixing the ~23 documented bugs is the single highest-leverage next step for coverage, and each fix also converts currently-dangling crash inputs into legitimate regression tests.Commit structure
72 commits, grouped for reviewability:
check:— new tests in thematic batches (subject lines state the area and test count, e.g. "check: 311 coverage tests for compiler, preprocessor and runtime error messages").sources:— 6 comment-only commits adding lcov exclusion markers (each with a one-line justification per span; no executable line touched).fix:— the 2 real bug fixes, each paired with its regression tests.check:commit removes tests that pinned buggy output, and one hardens the suite for all upstream CI configurations.Workload
4 hours of initial vibe coding for set-up (HITL) + 80 hours of looping (AFK) + 6 hours vibe debugging (HITL)
Energy cost: ~320 kWh
Tokens spent: ~1.2B within 4 days