[code-genetics] feat: resolve conflicting origins to detect the ultimate origin of files #1930

CHANGELOG.rst

@@ -10,6 +10,7 @@ Unreleased
 
 * feat: expose the ``is_addon`` attribute in the pipelines REST API
 * feat: [code-genetics] determine the origin of files from combined origin matches #1929 by @tdruez
+* feat: [code-genetics] resolve conflicting origins to detect the ultimate origin of files #1930 by @tdruez
 
 v38.2.0 (2026-10-22)
 --------------------

docs/application-settings.rst

@@ -381,6 +381,13 @@ API key using ``MATCHCODEIO_API_KEY``::
 
     MATCHCODEIO_API_KEY=insert_your_api_key_here
 
+.. note::
+    MatchCode.io does not report the path of a matched file in its package.
+    The :ref:`pipeline_find_origins` pipeline fetches this path from PurlDB when
+    :ref:`scancodeio_settings_purldb` is configured. Without it, the
+    ``containment`` rule of the :ref:`origin_determination` cannot apply to the
+    MatchCode.io matches.
+
 .. _scancodeio_settings_scanoss:
 
 SCANOSS

docs/built-in-pipelines.rst

@@ -150,6 +150,10 @@ Find Origins (addon)
     :ref:`scancodeio_settings_scanoss` to configure access to MatchCode.io and
     SCANOSS in your ScanCode.io instance.
 
+.. seealso::
+    :ref:`origin_determination` for the rules that determine the origin of each
+    file and directory.
+
 .. autoclass:: scanpipe.pipelines.find_origins.FindOrigins()
     :members:
     :member-order: bysource

docs/data-models.rst

@@ -63,6 +63,9 @@ OriginMatch
 
 OriginDetermination
 -------------------
+.. seealso::
+    :ref:`origin_determination` for the statuses and the values of ``rule``.
+
 .. autoclass:: scanpipe.models.OriginDetermination()
     :members:
     :undoc-members:

docs/index.rst

@@ -124,6 +124,7 @@ Indices and tables
     tutorial_web_ui_scan_repo_health
     scanpipe-concepts
     built-in-pipelines
+    origin-determination
     custom-pipelines
     scanpipe-pipes
     project-configuration

docs/origin-determination.rst

@@ -0,0 +1,170 @@
+.. _origin_determination:
+
+Origin Determination
+====================
+
+The :ref:`pipeline_find_origins` pipeline stores the candidate origin packages
+reported by each origin detection tool as origin matches, see
+:ref:`data_models_origin_match`. Its last step combines these matches into one
+origin determination per matched file or directory, see
+:ref:`data_models_origin_determination`.
+
+Each determination has a status and the name of the rule that set it. Only an
+agreement between two tools confirms an origin automatically. Any other case
+needs a review, with a proposed origin when a rule can select one.
+
+Candidates
+----------
+
+The origin matches of a file or directory are grouped by package into
+candidates:
+
+- The matches of the same package version are one candidate, even when the tools
+  report different PURLs, such as ``pkg:github/benjaminp/six@1.17.0`` and
+  ``pkg:pypi/six@1.17.0``, as SCANOSS reports the equivalent PURLs of a
+  component. A version with a ``v`` prefix, such as the ``v1.0.3`` Git tag, is
+  the same as the ``1.0.3`` release.
+- The snippet matches covering fewer than 20 lines of the file are ignored, such
+  as a common license header found in many unrelated packages. They are still
+  stored as origin matches.
+- When nested directories are matched to the same package, the deepest
+  directory is kept as the most precise match.
+- A file matched only to the package of its closest matched directory has no
+  determination of its own: its matches count for the directory.
+
+Statuses
+--------
+
+``auto-confirmed``
+    Set by the ``confirmed-by-two-tools`` rule.
+
+``needs-review``
+    Set by any other rule. The PURL is the proposed origin, or empty when no
+    candidate is preferred.
+
+``confirmed``
+    Set by a user who confirms the origin.
+
+``no-origin``
+    Set by a user when the file or directory has no open source origin.
+
+Rules
+-----
+
+The rules are evaluated in this order, and the first rule that applies sets the
+determination.
+
+.. list-table::
+    :header-rows: 1
+    :widths: 20 50 15 15
+
+    * - Rule
+      - Applies when
+      - Status
+      - Proposed origin
+    * - ``containment``
+      - Different packages: the copy in all the candidates but one is in a file,
+        or in a directory just under a vendor directory, named after another
+        candidate, such as ``pip/_vendor/six.py`` in pip. The vendor directories
+        are ``_vendor``, ``vendor``, ``vendored``, ``packages``,
+        ``third_party``, and ``extern``.
+      - ``needs-review``
+      - The remaining candidate.
+    * - ``highest-similarity``
+      - Different packages: one candidate is more similar than all the others by
+        at least 20 points, and all the similarities are known.
+      - ``needs-review``
+      - The most similar candidate.
+    * - ``earliest-release``
+      - Different packages: one candidate was released before all the others,
+        and all the release dates are known.
+      - ``needs-review``
+      - The earliest released candidate.
+    * - ``conflicting-candidates``
+      - Different packages, not resolved by the previous rules.
+      - ``needs-review``
+      - None.
+    * - ``preferred-version``
+      - Versions of the same package: one is more similar than all the others by
+        at least 20 points, or else only one is a release, as opposed to a commit
+        hash or a missing version, or else one release is more similar than the
+        other releases by at least 20 points.
+      - ``needs-review``
+      - The preferred version.
+    * - ``conflicting-versions``
+      - Versions of the same package, without a preferred version.
+      - ``needs-review``
+      - None.
+    * - ``no-exact-match``
+      - One candidate, with partial matches only: snippet, approximate, or
+        directory matches.
+      - ``needs-review``
+      - The candidate.
+    * - ``single-tool``
+      - One candidate, with an exact file or package match, reported by a single
+        tool.
+      - ``needs-review``
+      - The candidate.
+    * - ``confirmed-by-two-tools``
+      - One candidate, with at least one exact file or package match, reported by
+        two tools.
+      - ``auto-confirmed``
+      - The candidate.
+
+Similarity and release date
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+- The similarity is the percentage of the file lines matched: the SCANOSS
+  ``matched`` value, 100 for a MatchCode.io exact match, and the line coverage of
+  the MatchCode.io snippet matches. It is unknown for the directory and
+  approximate matches. The tools measure the same copy a few points apart, hence
+  the 20 points margin.
+- The release date is reported by SCANOSS, and by MatchCode.io when PurlDB
+  knows it. A copy cannot be older than its origin.
+- The similarity is checked before the release date, as it is measured on the
+  copy itself, while a release date only tells that a package version could
+  have been copied.
+
+Re-running the pipeline
+-----------------------
+
+- The automatic determinations, ``auto-confirmed`` and ``needs-review``, are
+  replaced.
+- The user determinations, ``confirmed`` and ``no-origin``, are kept. A
+  directory determined by a user covers its content: the origin matches under it
+  are not used.
+
+Origin packages
+---------------
+
+The files and directories of the ``auto-confirmed`` and ``confirmed``
+determinations are assigned to the package of their PURL. The files of a
+determined directory are assigned with it, unless they have a determination of
+their own.
+
+- A package detected by the codebase scan is reused, also when its PURL has
+  other qualifiers, such as a Maven ``classifier``.
+- Otherwise, the package is created from the tools data: download URL, homepage
+  URL, release date, and declared license. It is identified with the
+  ``origin_determination`` datasource ID, and created again on each run.
+
+Limitations
+-----------
+
+- A single tool never confirms an origin automatically, as it can miss other
+  candidates: SCANOSS reports one component per file, and MatchCode.io only
+  knows the indexed packages.
+- MatchCode.io rarely reports a release date, so the ``earliest-release`` rule
+  rarely applies to its candidates.
+- The ``earliest-release`` rule compares the release dates of the matched
+  versions, not of the first version containing the file. With an incomplete
+  index, the package that vendors a copy can be dated before the indexed
+  version of the original package.
+- The 20 lines threshold is calibrated on the MIT license header. A longer
+  header, such as the BSD-3-Clause one, still counts as a snippet match.
+- A small file copied with changes, matched as a snippet on fewer than 20
+  lines, is ignored.
+- The ``containment`` rule needs the path of the copy in each candidate.
+  SCANOSS only reports the file name, and the path of a MatchCode.io match is
+  fetched from PurlDB, when configured. When a PurlDB request fails, the paths
+  are partial and the rule does not apply to the files without a path.

scanpipe/pipelines/find_origins.py

@@ -25,6 +25,7 @@
 from scanpipe.pipelines import Pipeline
 from scanpipe.pipes import matchcode
 from scanpipe.pipes import origin
+from scanpipe.pipes import purldb
 from scanpipe.pipes import scanoss
 
 
@@ -39,7 +40,9 @@ class FindOrigins(Pipeline):
     Supported tools:
 
     - MatchCode.io: the codebase scan, including file and directory
-      fingerprints, is sent to the configured MatchCode.io server.
+      fingerprints, is sent to the configured MatchCode.io server. When PurlDB is
+      configured, the paths of the matched files in their package are fetched
+      from PurlDB.
     - SCANOSS: the codebase files are fingerprinted locally, then their paths,
       checksums, sizes, and fingerprints are sent to the configured SCANOSS
       server. The file contents are not sent.
@@ -77,6 +80,12 @@ def find_matchcode_origin_matches(self):
             )
             self.log(f"{origin_match_count:,d} MatchCode.io origin matches processed.")
 
+        if not purldb.is_configured():
+            self.log(
+                "PurlDB is not configured: the paths of the files matched by "
+                "MatchCode.io in their package are not available."
+            )
+
     def find_scanoss_origin_matches(self):
         """Scan the codebase with SCANOSS and store the reported origin matches."""
         if not scanoss.is_configured():
@@ -106,7 +115,9 @@ def determine_origins(self):
         Determine the origin of each matched file and directory from the origin
         matches of all the tools. An origin is confirmed automatically when two
         tools report the same package, with at least one exact match. Otherwise,
-        the origin needs a review. The origins confirmed by a user are kept.
+        the origin needs a review, with a proposed origin when a rule can select one
+        among the candidates, such as the candidate vendored by another candidate.
+        The origins confirmed by a user are kept.
         The files and directories of a confirmed origin are assigned to the origin
         package, created from the tools data when the codebase scan did not
         detect it.