CPG Layer: Joern Integration¶
This document records exactly what was verified about Joern as applied to our corpus, and which conclusions follow for the pipeline. It was compiled from primary sources (the joernio/joern source tree, official documentation), not from memory: every fact carries a confidence level.
1. The key discovery that changes the preconditions¶
php2cpg does not contain its own PHP parser. It invokes an external php with a bundled php-parser.phar.
Confirmed from the sources:
joern-cli/frontends/php2cpg/README.md: «A PHP to CPG converter based onphp-parse. Requirements: PHP (>=7.0)»;PhpParser.scala, methodphpParseCommand:Seq("php", "--php-ini", phpIniPath, phpParserPath) ++ Seq("--with-recovery", "--resolve-names", "--json-dump") ++ filenames;Php2Cpg.createCpg→isPhpVersionSupportedrunsExternalCommand.run(Seq("php", "--version"))and requires ≥ 7.1.0; otherwiseFailure("php not found or version not supported").
Consequence for the pipeline: without php in PATH the frontend does not degrade — it refuses. That is why cpg.py has a separate find_php() check, and the process environment is assembled so the directory containing php comes first in PATH — the frontend invokes literally php; a full path to php.exe does not work.
Translation note: the original stated «on this machine: PHP 8.5 found (
C:\tools\php85\php.exe)» — machine-local state of the author's environment, not a product requirement. The product requirement is: aphp≥ 7.1 executable discoverable inPATH(or viaRAGRAF_PHP_BIN/config — see roadmap R-CORE-2; the shipped code currently also probes two hardcodedC:/tools/...paths, a known defect).
2. Verified facts about structure and launch¶
| Fact | Source | Level |
|---|---|---|
Frontends and their scripts live in joern-cli/frontends/<name>/ |
joern-cli/build.sbt: frontendMappings("php2cpg", …) → ref -> s"frontends/$frontendName/$name" |
confirmed |
The joern, joern-parse, joern-export, joern-slice utilities are in joern-cli/bin/ |
sbt-native-packager mechanics (JavaAppPackaging) |
inference, not a quote |
The language is set with --language PHP (case-insensitive) |
frontends table; buildLanguageList — «Available languages (case insensitive)» |
confirmed |
--frontend-args passes arguments to the frontend verbatim |
CpgBasedTool.scala: ARGS_DELIMITER = "--frontend-args" |
confirmed |
--exclude <path> excludes files (path relative to the input) |
docs.joern.io/frontends/x2cpg | confirmed (but see §8 item 2: php2cpg rejects it) |
Script mode requires @main def exec(...) and --param k=v |
docs.joern.io/interpreter | confirmed (unusable on JDK 21 — see §8 item 3) |
A query result is written to a file with the #> operator |
docs.joern.io/shell | confirmed |
PHP_PARSER_BIN overrides the parser path |
PhpParser.scala: private val PhpParserBinEnvVar = "PHP_PARSER_BIN" |
confirmed |
The frontend always passes --with-recovery |
phpParseCommand |
confirmed |
Not verified (requires a run on the unpacked distribution):
- the exact contents of joern-cli/bin and the presence of .bat twins;
- whether joern has a single-command flag (-c/--command) — the documentation lacks it, so the pipeline does not rely on it;
- the behavior of -J-Xmx specifically in joern-parse.bat.
3. How to extract the data (verified against the exporter's code)¶
cpg.bin is a flatgraph binary; it cannot be read from Python directly. An export is needed.
| Need | --repr |
Compatible formats |
|---|---|---|
| CFG / DDG / CDG / PDG | cfg/ddg/cdg/pdg |
dot only |
| The whole graph with all edge types | all |
graphml, graphson, neo4jcsv, dot |
| Subgraph per method | cpg |
graphml, graphson, neo4jcsv, dot |
From JoernExport.scala: in exportWithFlatgraphFormat there are branches for exactly All and Cpg; for the rest — throw new NotImplementedError(...). That is, the combination --repr=pdg --format=graphml does not work.
Decision taken: the primary path is CPGQL scripts producing JSON. This gives exact control over what we extract (we need calls and methods, not the whole graph) and requires no GraphML/DOT parsing. The scripts live in src/ragraf/cpg.py (QUERY_CALLS, QUERY_METHODS, QUERY_TYPES).
Translation note: on JDK 21 the script mode proved broken (§8 item 3) and the shipped pipeline uses
joern-exportwith--repr=allinneo4jcsvformat, parsed bysrc/ragraf/joern_export.py. The CPGQL constants remain for reference/compatibility.
Additionally: joern-export itself builds the data-flow overlays (addDataFlowOverlayIfNonExistent calls OssDataFlow), so a separate overlay run for the PDG is unnecessary.
4. php2cpg limitations critical for our corpus¶
4.1. Syntax is limited to PHP 8.2¶
The bundled parser is nikic/PHP-Parser branch 4.x:
MODULE.bazel:PHP_PARSER_VERSION = "4.15.10";- fork README: «This is a PHP 5.2 to PHP 8.2 parser written in PHP».
The pilot is PHP 8.4. New constructs (property hooks, asymmetric visibility) are absent from the 4.x grammar → syntax errors.
Mitigating factor: the frontend always passes --with-recovery, so an incomplete AST is expected rather than a crash. How incomplete — verified on the run.
What to do: count this in the completeness metric. The report gets a separate line about files php2cpg failed to parse (in the logs these are Failed to process '<file>' — a warning under which the file silently drops out; such drops must be caught).
4.2. No composer.json → incomplete name resolution¶
buildFiles looks only for composer.json: SourceFiles.determine(config.inputPath, Set(".json"), …).filter(_.endsWith("composer.json")). The pilot has no composer.json (framework-less PHP, autoloading via spl_autoload_register).
Mitigating factor: the frontend walks the code twice — SymbolSummaryPass collects symbols across all namespaces, then AstCreationPass builds the AST with that summary («parses the code twice… This helps resolve symbols during the latter parse»). Intra-project links partially resolve.
4.3. JDK 21 is outside the tested range¶
docs.joern.io/installation: «JDK 19 … newer JDK versions might work, but have not been properly tested». We run JDK 21.
A separate warning in the slicing documentation: «Class path issues related to Circe are known to rear its head on Java versions later than 17» — concerns joern-slice specifically.
What to do: if joern-slice falls over on JDK 21, the alternative is slicing by our own means through CPGQL (CFG/DDG traversal), or installing JDK 19. This is resolved on the first run.
4.4. There is no incremental CPG update¶
Neither joern-parse, nor joern-export, nor the documentation describes an incremental rebuild; joern-scan --overwrite is described as full regeneration.
Consequence: the CPG rebuilds entirely on every reindexing. This agrees with the project principles (the index is immutable within a revision; reindexing is a separate act) but means the CPG stage is the most expensive in the run, and its time must be measured.
5. Linking the CPG to our graph¶
| CPG node | Key properties | Join key with our nodes |
|---|---|---|
FILE |
name |
source path |
METHOD |
name, fullName, filename, lineNumber, lineNumberEnd, isExternal |
(filename, lineNumber, name) |
TYPE_DECL |
name, fullName, filename, lineNumber |
classes, interfaces, traits, enums |
CALL |
name, code, methodFullName, dispatchType, lineNumber, columnNumber |
methodFullName — the thing tree-sitter's index does not have |
IDENTIFIER, LITERAL, LOCAL, PARAMETER |
name, code, lineNumber, typeFullName |
variables and values |
dispatchType distinguishes STATIC_DISPATCH from DYNAMIC_DISPATCH — a ready-made marker for separating proven from heuristic links.
Implemented name resolution (Indexer._make_fqn_resolver): exact full-name match → method-name-and-file match → refusal. The third outcome is principled: in framework-less PHP, short names (get, run) belong to dozens of classes, and binding «by name» would produce exactly the invented links that architecture principle P3 protects against.
Not verified: php-parser's line-numbering base (1-based or 0-based). Checked by comparison on the first run; on an off-by-one, the join key needs a correction.
6. What remained to be resolved by running¶
| Question | How it is checked |
|---|---|
| The presence and exact names of executables in the distribution | scripts/install-joern.ps1 (does this automatically) |
Whether joern-parse --language PHP works on our corpus |
the first run of the CPG stage |
| How many files php2cpg fails to parse because of PHP 8.4 | comparison against the Joern log and the tree-sitter result (we have 0 parse errors) |
| CPG build time on 112 K lines of PHP | timing of the cpg stage |
Memory consumption and the needed -Xmx |
timing + observation |
Whether the CPG's lineNumber matches our lines |
manual comparison of 20 methods |
Whether joern-slice works on JDK 21 |
a separate run |
7. State at the time of writing¶
Translation note: this is a historical snapshot of the author's machine (October 2026), kept for provenance. None of these components ships with RAGRAF; install them via
scripts/install-joern.ps1.
| Component | State (author's machine) |
|---|---|
| JDK 21 (Temurin 21.0.12.1) | ✅ installed in .tools/jdk, launch-verified |
| PHP CLI 8.5 | ✅ found — the php2cpg precondition is met |
| Joern 4.0.99 | ✅ installed in .tools/joern-cli (1633 MB), php confirmed in --list-languages |
The CPG layer in the pipeline (src/ragraf/cpg.py) |
✅ works: capability=full on the pilot |
Export parsing (src/ragraf/joern_export.py) |
✅ neo4jcsv via ijson |
The cpg stage in the indexer |
✅ merging confirmed calls into our graph |
| Install script | ✅ scripts/install-joern.ps1 |
8. What only a run revealed (facts absent from the documentation)¶
This is the most valuable part: Joern's documentation describes none of these points, and without them the CPG layer does not work.
| # | Discovered | How it manifested | Resolution |
|---|---|---|---|
| 1 | php2cpg requires an external php in PATH (it has no parser of its own) |
without PHP the frontend does not start at all | find_php() + putting the php directory first in PATH |
| 2 | php2cpg does not accept --exclude |
Unknown option --exclude → RuntimeException: unable to parse XTypeStubsParserConfig |
filtering service paths AFTER the export (is_noise_file) |
| 3 | Script mode (--script) does not work on JDK 21 |
compilation error: no predef script (importCpg, cpg not found) |
moved entirely to joern-export |
| 4 | GraphSON export dies on memory even at 18 GB | OutOfMemoryError: Required array length 2147483640 + 8 is too large |
neo4jcsv format: streaming CSVs, 223 MB, 27 s |
| 5 | --repr=cpg creates a file per method and crashes |
731 files, 739 MB, FileSystemException |
--repr=all + filtering while parsing |
| 6 | JAVA_OPTS does not reach the JVM |
OutOfMemoryError at any value of the variable |
the limit is passed as the argument -J-Xmx18g (the form without =) |
| 7 | Joern paths do not match ours | leading dot and backslashes; matching always refused | normalize_joern_path |
| 8 | Joern writes methods with ->, not :: |
App\Core\Config->get matched nothing |
normalize_joern_name; result: 411 links by full name |
| 9 | AST traversal without a method boundary captures everything | the <global> stub exists in every file and contains the whole file: 179 330 of 183 357 calls got it as owner |
stop traversal at nested METHODs |
| 10 | In the Joern export the vertex array is called vertices, not nodes |
parsing yielded 0 vertices against 83 542 parsed edges | key @value.vertices.item; properties live in properties under UPPER-CASE names |
| 11 | Most CALLs are language operators | 50 K of 183 K are <operator>.assignment |
_is_noise_call; 17 822 real calls remain |
| 12 | PHP builtins must be classified separately | 13 008 of 17 822 «unresolved» were array_push, is_array, Iterator.next |
_is_builtin_target: 7 819 recognized as normal; the honest remainder — 4 985 |
Lesson for the project. Of the 12 items, 7 led to a silent loss of data: the code did not crash, but the result turned out zero or distorted. Such defects are caught only by cross-checking independent paths — exactly as happened with CPGQL (worked in theory, not in practice) and with the <global> filter (formally the traversal is correct; in fact almost all data landed on one owner).
9. Result on the pilot¶
| Indicator | Value |
|---|---|
capability |
full (CPG connected) |
| Full indexing time with CPG | 71 s (of which CPG — 64 s) |
| CPG build | 18.6 MB, ~50 s for 323 files |
| Export | 223.8 MB, 132 files, ~27 s |
| Meaningful vertices extracted | 193 442 |
| Edges extracted | 3 348 516 (incl. 444 464 CFG, 930 567 data-flow) |
| Calls after filtering | 17 822 of 183 357 |
| Links confirmed in the graph | 603 (411 by full name, 192 by name and file) |
| PHP builtins recognized | 7 819 |
| Honestly unresolved | 4 985 |
slice |
works (was unavailable in degraded mode) |
| Effect on M2 | recall 0.771 → 0.858; median 1.000; zero-recall cases 0 (was 1); full connectivity in 13 of 20 (was 11) |
Translated from the original Russian integration study. Section 8 is the operational core of this document and is translated verbatim in structure; machine-local state is annotated rather than removed.