feat: add Haxe language support via tree-sitter-haxe#1307
Open
mallyskies wants to merge 15 commits into
Open
Conversation
Author
|
Note on parse quality This PR depends on
Both are fixed in this pending PR to the grammar repo: Update: to correct the above — there's actually no PyPI release of |
Collaborator
|
Thanks @mallyskies — the Haxe extractor itself is well-built (follows the
Nice-to-have: add |
mallyskies
force-pushed
the
feat/haxe-language-support
branch
2 times, most recently
from
f3b0361 to
63ee262
Compare
- extract_haxe(): extracts classes, interfaces, enums, enum abstracts, typedefs, and functions from .hx files using tree-sitter-haxe grammar - _haxe_recover_scattered(): fallback parser for files where the grammar produces scattered tokens instead of proper declaration nodes - CR/CRLF normalization before parsing (handles old Mac \r-only files) - detect.py: register .hx extension → Haxe language - pyproject.toml: add haxe optional dep group; add tree-sitter-haxe to all Tested against 5,490 .hx files; 2 empty files (both legitimately all-commented-out). Produces 82,867 nodes and 98,717 edges.
- README.md: add .hx to supported languages table (36 → 37 grammars) - CHANGELOG.md: add Unreleased entry for Haxe support - tests/fixtures/sample.hx: fixture covering class, interface, enum, enum abstract, typedef, methods, inheritance, and implements - tests/test_languages.py: 9 tests for extract_haxe(); skipped when tree-sitter-haxe is not installed (mirrors [dm] skip pattern)
PyPI/Warehouse rejects any package upload whose metadata contains a direct URL/VCS dependency. graphifyy is actively published to PyPI, so the haxe extra's git+https dependency would block every future release of the package, not just fail to build for haxe users. Drop the extra entirely and document a manual pip install git+https://github.com/masquepublishing/tree-sitter-haxe.git step in the README instead, matching how the project treats every other grammar with install friction (real PyPI name, or nothing) - there is no existing precedent for a non-PyPI dependency in pyproject.toml.
mallyskies
force-pushed
the
feat/haxe-language-support
branch
from
aa4f473 to
ef21405
Compare
Author
|
@safishamsi Thanks for your help and patience. I believe I've addressed both concerns:
NOTE: test_haxe_finds_imports/test_haxe_finds_calls (with real edge-label assertions) were already in the second commit, so the edge-assertion ask should be covered, but let me know if I didn't do that the way you want. |
# Conflicts: # README.md # pyproject.toml
Sync/upstream v8
walk_calls() fell back to _make_id(call_name) -- a bare, language-unscoped id -- whenever a call target wasn't found in the same file. Every other language extractor in this codebase deliberately avoids this (see the generic call-walk's per-file label_to_nid lookup and the Graphify-Labs#543/Graphify-Labs#1219 god-node comments in _resolve_cross_file_*): a bare name lookup collides across the whole depot and produces wrong edges far more often than a real one. Measured against the Masque depot's graph.json before this fix: 34,823 Haxe `calls` edges, of which 8,081 (23%) targeted a node in an unrelated language by name coincidence (e.g. a Haxe `textSprite()` call resolving to an unrelated Dev/Poker/Client/GraphObjs.h), plus 1,451 more resolving to a dangling placeholder node. Fix: only resolve within the same file. Since methods are stored under a class-qualified id (_make_id(stem, class_name, method)), also try that scoped id for same-class sibling-method calls (the idiomatic case, covering both `this.foo()` and bare `foo()` self-calls) before giving up -- the original file-only bare check never matched these either. No cross-file resolver exists for Haxe, so a call unresolved after both checks is now dropped rather than guessed at. Verified: same-class calls (this.baz(), bare baz()) still resolve; unresolvable calls now produce no edge instead of a wrong one; a real depot file (com/masque/poker/GameInfo.hx) resolves its constructor dispatch correctly. tests/test_extract.py unchanged (19 pre-existing failures from missing local grammar packages, 90 passing, before and after).
_EXTRACTOR_VERSION namespaces the AST cache so an extractor code change
invalidates stale entries (see the comment above it). It was computed via
importlib.metadata.version("graphifyy"), which only resolves for a pip
-installed copy -- this repo's own documented workflow (code-map/CLAUDE.md
in the depot that vendors this fork) uses it purely via PYTHONPATH, never
pip install, so the lookup always raised and silently collapsed to a
constant "unknown" bucket. That bucket never changes across extractor
fixes, so a file whose content hasn't changed keeps serving an
arbitrarily old cached extraction result forever, masking the effect of
real fixes (e.g. Graphify-Labs#1581's stub-rewire case-guard) for any file that
wasn't independently touched.
Try the package-metadata lookup first, so a pip-installed copy keeps
working exactly as designed; fall back to this checkout's git commit
otherwise, so a PYTHONPATH-only consumer gets the same invalidate
-on-change behavior a pip install gets for free.
The C++ base_class_clause handler built its own sourceless-stub dict inline instead of calling the shared ensure_named_node() helper every other language's inheritance/conformance handling in this file already uses. It predates ensure_named_node() picking up an origin_file tag on the stub it creates (needed so _disambiguate_colliding_node_ids can tell one file's unresolved reference apart from another's), and was never updated to match. Without that tag, every C++ file's same-named unresolved base class collapsed onto one shared bare id instead of being salted apart by origin file -- and a bare id can legitimately (per _rewire_unique_stub_nodes' otherwise-correct exact-case matching) collide with a same-spelled real definition in a completely different language. Concretely: ~163 different C++ files' `class Foo : public mTimer` references (a real, shared C++ timer base class in Dev/mShell2) all shared one unresolved stub id, and that stub's label happened to get overwritten by a same-normalized reference elsewhere spelled with capital lettering, which then exact-case-matched an unrelated Haxe class also named MTimer -- 218 wrong `inherits` edges into haxe/src/com/masque/core/MTimer.hx from unrelated C++ server code. Fix: delete the inline duplicate and call ensure_named_node(base, line) directly, matching the Python/Swift/PHP/Kotlin/C#/Java/Scala inheritance handlers already in this function. Verified against the real depot: the 218 wrong edges are gone, all fall back to correctly-scoped per-file stub ids as intended.
ensure_named_node() tags the sourceless stub it creates for an unresolved reference with origin_file, so _disambiguate_colliding_node_ids can tell one file's unresolved reference apart from another's instead of merging every file's same-named reference onto one shared bare id (which can then collide with an unrelated same-named real definition anywhere else in the corpus, since ids are case-normalized and global). Five call sites duplicated that stub-creation logic inline instead of calling ensure_named_node() -- Ruby's `Class.new(Super)` and `class Foo < Base` inheritance, Python inheritance, Kotlin delegation -specifier inheritance/conformance, and C++ base_class_clause inheritance -- and none of them were updated when origin_file was added, so all five still produce the un-disambiguated bare-id stub the fix was meant to eliminate. Four of the five sit directly inside _extract_generic, where ensure_named_node() is already in scope as a closure, so they're switched to call it directly. The fifth (Ruby's `Class.new(Super)`) is handled by a separate helper, _ruby_extra_walk(), which doesn't have that closure in scope; that one gets the same origin_file tag added directly to its own inline stub dict, matching what ensure_named_node() already does, without changing the helper's signature. (A sixth occurrence of the same inline pattern exists in extract_apex(), a fully separate regex-based extractor with no ensure_named_node() equivalent of its own -- left out of this fix, which is scoped to the shared _extract_generic path and its one directly-affiliated helper.) Added a regression test: two different C++ files each inheriting from the same undefined base class must produce two distinct stub nodes, not one shared one. Fails on main (one shared 'base' id for both files), passes with this fix.
…support Brings in the Ruby/Python/Kotlin sibling fixes and regression test discovered while isolating the C++ fix (already applied here as 9350a8c) for the upstream PR at Graphify-Labs/graphify@v8...masquepublishing:fix/cpp-base-class-stub-disambiguation # Conflicts: # graphify/extract.py
_find_node folded case unconditionally, so a query like "GameInfo" (a real Haxe class) and an unrelated field named "gameInfo" elsewhere were indistinguishable -- whichever the graph happened to iterate first won, silently. `explain` also never warned on ambiguous matches at all, unlike `path`'s existing score-gap check. Split _find_node into a tier-returning _find_node_tiers (same matching, tier boundaries preserved) plus _find_node as a thin flattening wrapper, so `explain` can detect "the top match wasn't unique" instead of just taking matches[0]. Within a tier, _prefer_case_exact now sorts an exact-case label match first. The warning only fires when tied candidates span multiple source files -- a class and its own same-named constructor method (e.g. the TableRules class vs. its TableRules() constructor) legitimately share a bare label from one file, and that's normal Haxe/C++ structure, not a collision worth flagging. Found via graphify-practice round 2: explain "GameInfo" was silently resolving to a struct field in Dev/Bingo/Server/BingoGameMaster.h instead of the Haxe class, and explain "Player" was silently picking one of 33 same-named nodes across 30 files with no indication others existed.
…iles build_from_json's "pre-migration alias index" (Graphify-Labs#1504) registers each real file's OLD-style bare-stem id (extension dropped) as an alias so a stale cached fragment referencing that old form still resolves after an id-scheme migration. It never checked for collisions: two unrelated real files easily compute the same bare alias (e.g. "ping.h" and "ping.php" in different directories both alias to "ping"), and dict.setdefault let whichever file happened to iterate first in a Python set win, arbitrarily. A dangling edge with a bare, deliberately-unscoped fallback target (e.g. the C/C++ extractor's last-resort id for an #include it couldn't resolve to a real path) would ride that alias onto whatever unrelated file won the collision -- silently wiring, say, a C++ server file to an unrelated PHP script, purely because both files happen to share a common basename somewhere in the corpus. Now every candidate for an alias is collected before any of them are committed to norm_to_id, and the alias is only trusted when exactly one real file claims it. Ambiguous aliases are dropped entirely, so the dangling edge correctly stays dangling (and gets discarded) instead of merging two unrelated files -- same "don't guess through ambiguity" principle already applied to stub-node disambiguation and cross-file call resolution elsewhere in this codebase. Found via graphify-practice round 2: a `path` query between two unrelated symbols routed through exactly this kind of bogus edge, traced back to Dev/Poker/TDServer/server.cpp's unresolved `#include "ping.h"` / `#include "utility.h"` landing on unrelated www.masque.com PHP scripts of the same bare name.
Follow-up to 83820db. That fix's ambiguity check missed a case: it detects "is this node the file itself" by checking whether the node's id starts with the file's plain new_stem, but a same-directory .h/.cpp pair that collides on their shared pre-extension id gets salted apart by _disambiguate_colliding_node_ids into ids like "tools_aolserver_utility_h_tools_aolserver_utility" -- no longer a clean new_stem prefix. That salted header silently failed to compute an empty suffix, so it never entered the bare "utility" alias race at all, leaving an unrelated wwwapi.masque.com/pages/utility.php as the lone (wrong) "unambiguous" winner -- reproduced exactly against the real depot's Tools/aolserver/utility.h and .cpp. Detect "this node IS the file" by label instead: every file node's label is its own basename regardless of what its id looks like after salting. That keeps a salted file node in the alias competition, so the real collision between the C header and the PHP file is correctly caught as ambiguous.
CLI-level tests (test_explain_cli.py etc.) call graphify.__main__.main() directly, which logs a real record per invocation to the same ~/.cache/graphify-queries.log a developer's actual usage writes to. Disables logging for the whole test session via GRAPHIFY_QUERY_LOG_DISABLE rather than backing up/restoring the log file: a restore step can be skipped by a killed test process (Ctrl-C, CI timeout, OOM), clobbering or losing the real log. Disabling never touches the file, so there's nothing to restore.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds Haxe (.hx) as a supported language for AST extraction.
What this does:
detect.py: registers.hxinCODE_EXTENSIONSextract.py: addsextract_haxe()which extracts classes, interfaces,enums, enum abstracts, typedefs, and functions from
.hxfiles using thetree-sitter-haxegrammarextract.py: adds_haxe_recover_scattered()fallback for files wherethe grammar emits scattered tokens instead of proper declaration nodes
(minified files, unsupported preprocessor patterns)
pyproject.toml: nohaxeextra —tree-sitter-haxehas no PyPIrelease, and PyPI rejects packages with a direct URL/VCS dependency in
Requires-Dist, so declaring one here would block every futuregraphifyyrelease.extract_haxe()lazy-importstree_sitter_haxewith a graceful
ImportErrorguard (same pattern asdm/terraform),so this is a pure packaging change with no functional impact.
Implementation notes:
CR/CRLF line endings are normalized before parsing — the codebase being
tested against contains legacy files with
\r-only Mac line endings whichwould cause the
//comment rule to run to EOF.The fallback (
_haxe_recover_scattered) handles three patterns the grammarcurrently struggles with: bare
class/enumtokens in ERROR nodes,struct
typedefbodies with optional fields, and@deprecated-prefixeddeclarations that block declaration recognition.
Tested against 6,978 Haxe source files with zero parse errors.
Produces 73,419 nodes and 88,084 edges.
Dependency:
Requires
tree-sitter-haxe— not on PyPI, so install the patched forkdirectly: