Write the PHPDoc language down as a grammar, and fuzz from it

[ondrejmirtes]

doc/grammars now holds the language this library reads, written in the PP3 format the phplrt compiler reads: the tokens, the type language, the constant expressions and the PHPDoc itself, down to its Doctrine annotations. The rules are named after the methods of PhpDocParser, TypeParser and ConstExprParser they stand for and are written in the order those methods try things in, so the two can be read side by side.

It replaces type.abnf, phpdoc-param.peg and phpdoc-method.peg, none of which described the whole language and none of which anything read.

File	What it holds
lexemes.pp3	Every token the language is read into
common.pp3	What the grammars share: names, brackets, line breaks
types.pp3	The type language of TypeParser
const-expr.pp3	The constant expressions of ConstExprParser
phpdoc-block.pp3	The PHPDoc itself: its tags, its text, its Doctrine annotations
type.pp3, constant-expr.pp3, phpdoc.pp3	The three entry points

What the grammars describe

A PHPDoc that is written correctly. The parser reads a broken one as well, by turning whatever it cannot read into an InvalidTagValueNode carrying the very error it has raised, and a grammar has no way of writing that error down. So a place the parser raises an error at is written as something the grammar cannot recognize, and everything the grammars describe is something the parser has to read in full.

Most of those are written as a ! predicate forbidding whatever the error would have been raised on. A name followed by < has to go on into a generic type or a callable, because Foo< is an error rather than the type Foo followed by something else:

IdentifierAtomic
  : ...
  | !ShapeBrace() Identifier() !<T_DOUBLE_COLON> ( IdentifierSuffix() | !<T_OPEN_ANGLE_BRACKET> )
  ;

The same predicate is what keeps a rule from giving back what it has read. @template T of is an error rather than a template named T with the description of, so the bound is written as "either a bound or no of at all".

Three corners are left out on purpose, each written up where the rule that skirts it is written: a description that ends at a tag written in the middle of a line, the same on a line after the first (where the parser reads that line twice), and a tag whose value a rule reads written with a parenthesis after it.

The tokens are not read by phplrt

A grammar here is not read by the lexer it declares — it is read by the very tokens PHPStan\PhpDocParser\Lexer\Lexer produces, handed over by tools/phplrt/Fuzzer/TokenStream.php. The %token declarations therefore name the tokens and document the language without being what reads it, and TokenStream is what tells apart the things the lexer does not read as a token of its own: a word the parser compares by value (is, array, covariant, …), a tag whose value a rule of its own reads, a bracket or asterisk whose neighbouring whitespace decides what it means, a tag a space is written before, and a < opening what the parser recognizes as an HTML tag. Telling them apart there is what lets the grammars be written without semantic predicates, which PP3 has none of.

FuzzyTest

A grammar says what a PHPDoc may be written as, so it can be walked the other way round and asked for PHPDocs instead of being asked about one. tools/phplrt/fuzz.php compiles a grammar, walks its rules and writes down what comes out; FuzzyTest reads every one of them and asks for a type to come back as the very same type once it has been printed.

Two things are thrown away before an input is written down, and neither of them is the parser being wrong: one whose text does not read back as the very tokens it was written of, and one the grammar itself does not recognize (a walk over the rules says what the language is written of, while a parser reads the alternatives of a rule in the order they are written and stops at the first one that fits).

This replaces the abnfgen fuzzer, which needed a C program built from a tarball at tests/abnfgen-0.20.tar.gz and only covered the type language and the constant expressions. build-abnfgen.sh and the tarball are gone.

A whole PHPDoc is only asked to be read in full, not to survive being printed and read again: a description written across several lines is printed as it was read rather than as the lines of a PHPDoc, which 8 of the 318 PHPDocs in this project's own test corpus already run into.

Running it

The compiler asks for PHP 8.4, which this library still runs without, so it is a development dependency of its own under tools/phplrt and FuzzyTest skips itself wherever it is missing:

make grammars-install   # the toolchain

There is no corpus to keep and none to refresh when a rule changes: FuzzyTest runs fuzz.php itself, once per grammar, every time it runs, so what is generated changes the moment the grammar is read again. The inputs of the last run are left in temp/fuzzy to be looked at.

Only the new Grammars job of build.yml runs it for real, on PHP 8.4 and 8.5.

Rule coverage

Every .pp3 is read — phpdoc.pp3 includes all the others — and the three roots exist so TypeParser and ConstExprParser are driven directly and get the print round trip. Measuring which named rules the walk actually reaches, over 8000 accepted inputs per root:

root	named rules	never reached
type.pp3	71	7
constant-expr.pp3	18	0
phpdoc.pp3	130	11

Ten of the eleven are referenced only from inside a !/& predicate, which reads nothing, so the walk correctly never descends into them; the seven under type.pp3 are the array-literal rules that !ArrayLiteral() forbids a type from starting with, and they are fully covered from constant-expr.pp3. The one left over is covered from type.pp3.

What it found

9b0ba2c — the lookahead in Lexer separating the & of Foo &$bar from the & of Foo&Bar lists [.,=)], and the comment above it says the dot stands for TOKEN_VARIADIC. A single dot also starts a float written without its leading zero, so Foo & .5 lexed as a reference and the type ended at the & — while Foo & 5 and Foo | .5 both read fine. It is now \.\.\., which is what the comment already claims; Foo &...$bar, callable(Foo&), callable(Foo&, and callable(Foo&=) lex as before.

It was also what kept the printer from printing an intersection that can be read again: & is printed with no space around it, so Foo & .5[] came back out as Foo&.5[].

Happy to split that commit off into a PR of its own if you would rather have it separately.

And one bug in the grammar itself, found by measuring the coverage above. DoctrineTag was written as

DoctrineTag
  : DoctrineTagToken() DoctrineArguments()? DoctrineDescription()

so the grammar said @\Foo( with no closing parenthesis is a tag with no arguments whose description happens to begin with a (. The parser commits to reading arguments the moment a parenthesis follows the tag, and errors. The no-arguments branch now forbids it, the way GenericTag already did.

It had been invisible because the fuzzer never wrote a Doctrine tag at all: the literal it used, @Foo_Bar, does not lex as one — TOKEN_PHPDOC_TAG is tried first and matches @Foo, leaving _Bar as a name, so every input carrying one was thrown away by the re-lexing filter. A tag reads as a Doctrine one only where a \ or _ follows the @ straight away. With @\Foo instead, Doctrine annotations appear in about a tenth of the generated PHPDocs and the whole Doctrine section of the grammar is exercised for the first time.

The three fixes of #308 came out of the same work.

Verification

4741 tests, phpcs, PHP-Parser lint and PHPStan all clean. Beyond the suite, a sweep of 60 seeds — 180,000 generated inputs across the three grammars — reads every one in full, with the type and constant-expression ones surviving the print round trip.

🤖 Generated with Claude Code

https://claude.ai/code/session_01GavWEzia8ZVUYEe6JEde9i