data_validation: NER and gitleaks scans, SPDX/BCP 47 checks, NIST AI 100-2 IDs

.github/workflows/datasets-validate-all.yml

@@ -78,3 +78,53 @@ jobs:
 
       - name: data_validation tests
         run: python -m pytest data_validation/tests -v
+
+  # gitleaks (pinned release, checksum verified) over the data and tooling.
+  # Blocking: its rules target real credential formats, and the datasets'
+  # synthetic secrets use placeholders that it does not match.
+  secrets:
+    runs-on: ubuntu-latest
+    env:
+      GITLEAKS_VERSION: '8.30.1'
+      GITLEAKS_SHA256: '551f6fc83ea457d62a0d98237cbad105af8d557003051f41f3e7ca7b3f2470eb'
+    steps:
+      - name: Checkout
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+
+      - name: Install gitleaks
+        run: |
+          tarball="gitleaks_${GITLEAKS_VERSION}_linux_x64.tar.gz"
+          curl -sSfL -o "$tarball" "https://github.com/gitleaks/gitleaks/releases/download/v${GITLEAKS_VERSION}/${tarball}"
+          echo "${GITLEAKS_SHA256}  ${tarball}" | sha256sum -c -
+          tar -xzf "$tarball" gitleaks
+
+      - name: Scan datasets and data_validation for secrets
+        run: |
+          ./gitleaks dir datasets --redact --no-banner --exit-code 1
+          ./gitleaks dir data_validation --redact --no-banner --exit-code 1
+
+  # Named-entity scan for person and organisation names in the datasets that
+  # require anonymization. Warnings for reviewers; never fails the job.
+  ner-pii:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+
+      - name: Set up Python
+        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+        with:
+          python-version: '3.11'
+
+      - name: Install requirements
+        run: python -m pip install --quiet -r data_validation/requirements.txt -r data_validation/requirements-ner.txt
+
+      - name: NER scan (job summary)
+        run: |
+          {
+            echo '## Name detection (NER) in anonymized datasets'
+            echo
+            echo '```'
+            python data_validation/qc_tools/ner_pii_scan.py
+            echo '```'
+          } >> "$GITHUB_STEP_SUMMARY"

.gitignore

@@ -4,6 +4,7 @@
 *~
 node_modules/
 __pycache__/
+.pytest_cache/
 *.pyc
 .env
 venv/

data_validation/README.md

@@ -40,12 +40,15 @@ The tooling has two jobs: keep the research data behind OWASP deliverables (the
 |---|---|---|
 | `schema_validator.py` | Schema violations, for files no dataset validator covers | File with no schema to check against |
 | `dsgai_mapping_check.py` | DSGAI ID not in the taxonomy; a mapping value that is not a DSGAI ID | Same ID twice in one mapping; more than 6 mappings |
-| `crossref_validator.py` | Malformed CVE/GHSA/CWE/ATLAS ID; CVE year in the future; unknown CWE, ATLAS or OWASP Top 10 ID (`LLM01:2026`, `ASI01:2026`) | Deprecated CWE; retired ATLAS ID; OWASP ID without its edition year; malformed ID in free text (often a placeholder) |
+| `crossref_validator.py` | Malformed CVE/GHSA/CWE/ATLAS ID; CVE year in the future; unknown CWE, ATLAS, OWASP Top 10 (`LLM01:2026`, `ASI01:2026`) or NIST AI 100-2 (`NISTAML.018`) ID | Deprecated CWE; retired ATLAS ID; OWASP ID without its edition year; malformed ID in free text (often a placeholder) |
 | `severity_check.py` | CVSS v3.x/v4.0 vector that doesn't parse; score that doesn't match its vector; qualitative rating outside the score's FIRST band | Score with no vector |
 | `date_check.py` | Date that isn't ISO 8601; date in the future; documented, reported or observed after `date_added` | — |
+| `metadata_check.py` | Licence field that doesn't resolve to one SPDX licence (ID, expression or full name; the ID is preferred); language field that isn't a BCP 47 tag of registered IANA subtags | Deprecated SPDX ID or language subtag |
 | `anonymization_scanner.py` | — | Email addresses, keys and tokens, private or (in a network context) public IPs, internal hostnames, home paths, SSNs |
 | `dedup_checker.py` | Two records in one dataset with the same ID | Records with identical or ≥ 80% similar text |
 
+CI also runs **gitleaks** (pinned release, checksum-verified) over `datasets/` and `data_validation/`; a finding fails the PR.
+
 Anonymization and similarity hits are warnings because a pattern cannot tell a real secret from a synthetic one; the prompt-injection datasets contain fake credentials on purpose. Use obviously fake values to keep a file quiet: `example.com` and other RFC 2606 domains, the RFC 5737 documentation addresses (`192.0.2.0/24`, `198.51.100.0/24`, `203.0.113.0/24`), or placeholders such as `<API_KEY>` or `[API_KEY]`.
 
 CVE and GHSA IDs are format-checked only; there is no offline list to confirm they exist. Framework control IDs (ISO, NIST, CIS) are not checked: see [`reference_data/SOURCES.md`](reference_data/SOURCES.md).
@@ -72,6 +75,7 @@ After the automated checks pass, contributions are reviewed by cybersecurity and
 | `bias_report.py` | Markdown coverage report: records per dataset, a DSGAI × dataset matrix, DSGAI entries nothing maps to, and the spread of severity and each dataset's categorical fields |
 | `anomaly_detector.py` | A DSGAI entry most of a dataset maps to, records with far more mappings than usual, and one severity or category value dominating a dataset (datasets of 10+ records) |
 | `online_check.py` | Runs weekly (`datasets-online-checks` workflow), not on PRs, because it needs the network. **Fails** on a CVE the CVE Program has rejected or doesn't know, a withdrawn GHSA, a dead cited link (with the Wayback Machine snapshot to use instead), or a CVE in CISA KEV whose entry isn't marked `exploited_in_the_wild`. Reports every CVE's EPSS v4 score and KEV status. These change daily, so they're reported, not stored |
+| `ner_pii_scan.py` | Person names (and, in `incident_dataset`, organisation names) found by named-entity recognition (Microsoft Presidio + spaCy) in the datasets that require anonymization. Warnings only, in the CI job summary; reviewed false positives go in `reference_data/ner_allowlist.txt`. Optional dependencies: `requirements-ner.txt` |
 | `consistency_check.py` | **Fails** on internal references (`DSGAI-VULN-…`, `DSGAI-EXP-…`, `DSGAI-RA-…`) that point at no existing entry; notes CVEs cited by exploits that have no vulnerability entry |
 
 ---
@@ -115,13 +119,14 @@ data_validation/
 ├── qc_tools/                Reports for reviewers
 ├── schemas/                 Schemas for datasets without their own schema.json
 │                            (the rest are pointers to the dataset-local schemas)
-├── reference_data/          DSGAI, MITRE ATLAS, CWE and OWASP Top 10 lookup tables; see SOURCES.md
+├── reference_data/          DSGAI, MITRE ATLAS, CWE, OWASP Top 10, NIST AI 100-2, SPDX and
+│                            IANA language lookup tables; see SOURCES.md
 └── tests/                   Tests and deliberately valid/invalid fixtures
 ```
 
 Dataset-local schemas (`datasets/<name>/schema.json`) are authoritative. `schemas/exploit`, `vulnerability`, `riskassessment` and `agentdataflow_trace` are kept only as `$ref` pointers to them; `schemas/incident`, `rag`, `crossframework_mapping` and `promptinj_testcase` are the only schemas for their data.
 
-The ATLAS and CWE tables are generated from MITRE's releases by `reference_data/update_reference_data.py`; don't edit them by hand. The `reference-data-refresh` workflow runs it on the 2nd of each month and, when MITRE has published something new, pushes a branch and opens an issue for a maintainer to turn into a PR. `owasp_top10.csv` is maintained by hand when OWASP publishes a new edition.
+The ATLAS, CWE, SPDX and language tables are generated from their publishers' releases by `reference_data/update_reference_data.py`; don't edit them by hand. The `reference-data-refresh` workflow runs it on the 2nd of each month and, when MITRE has published something new, pushes a branch and opens an issue for a maintainer to turn into a PR. `owasp_top10.csv` and `nist_ai_100_2.csv` are maintained by hand when a new edition is published.
 
 ---
 

data_validation/qc_tools/ner_pii_scan.py

@@ -0,0 +1,124 @@
+"""
+Finds personal and organisation names that regex patterns cannot, using
+named-entity recognition (Microsoft Presidio on a spaCy English model).
+
+WARN only: NER is probabilistic, and a reviewer decides. It runs on the
+datasets whose contribution rules require anonymization:
+
+  incident_dataset                   person and organisation names (its
+                                     README forbids both)
+  riskassessment_dataset,
+  agentdataflow_toolexchange_traces,
+  rag_dataset                        person names
+
+To keep precision usable on technical text, a hit counts only when the
+model is at least 85% confident and the text looks like a proper name: two
+or more capitalised words ("Jane Doe", "Acme Corp"), never an acronym such
+as "PII" or "EU". Reviewed false positives go in
+reference_data/ner_allowlist.txt. The prompt-injection datasets are not scanned: they are
+mostly Turkish, and the English model would only produce noise.
+
+Optional dependencies, kept out of requirements.txt so the core checks
+stay single-dependency:  pip install -r requirements-ner.txt
+
+Usage:
+    python ner_pii_scan.py
+    python ner_pii_scan.py --dataset ../../datasets/incident_dataset/
+"""
+from __future__ import annotations
+
+import argparse
+import re
+import sys
+from pathlib import Path
+
+if __package__ in (None, ""):
+    sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
+
+from validators._common import (  # noqa: E402
+    DATASETS_ROOT, REFERENCE_DIR, WARN, Finding, ensure_utf8_stdout, iter_data_files,
+    load_json, print_findings, walk_strings,
+)
+
+CHECK = "ner-pii"
+MODEL = "en_core_web_sm"
+SCORE_THRESHOLD = 0.85
+MIN_TEXT_LEN = 15
+# Two or more capitalised words: a proper name, not an acronym or a single word.
+PROPER_NAME_RE = re.compile(r"^[A-Z][a-z]+(?:[ '-][A-Z][a-z]+)+$")
+ALLOWLIST_PATH = REFERENCE_DIR / "ner_allowlist.txt"
+SCOPE = {
+    "incident_dataset": ["PERSON", "ORGANIZATION"],
+    "riskassessment_dataset": ["PERSON"],
+    "agentdataflow_toolexchange_traces": ["PERSON"],
+    "rag_dataset": ["PERSON"],
+}
+
+
+def load_allowlist() -> set[str]:
+    """Reviewed false positives (reference_data/ner_allowlist.txt)."""
+    lines = ALLOWLIST_PATH.read_text(encoding="utf-8").splitlines()
+    return {line.strip() for line in lines if line.strip() and not line.lstrip().startswith("#")}
+
+
+def build_analyzer():
+    from presidio_analyzer import AnalyzerEngine
+    from presidio_analyzer.nlp_engine import NlpEngineProvider
+    config = {
+        "nlp_engine_name": "spacy",
+        "models": [{"lang_code": "en", "model_name": MODEL}],
+        "ner_model_configuration": {
+            "model_to_presidio_entity_mapping": {"PERSON": "PERSON", "ORG": "ORGANIZATION"},
+            "labels_to_ignore": [],
+        },
+    }
+    engine = NlpEngineProvider(nlp_configuration=config).create_engine()
+    return AnalyzerEngine(nlp_engine=engine, supported_languages=["en"])
+
+
+def scan_data(path: Path, data, entities: list[str], analyzer, allow: set[str] | None = None) -> list[Finding]:
+    allow = load_allowlist() if allow is None else allow
+    findings: list[Finding] = []
+    for pointer, _key, value in walk_strings(data):
+        if len(value) < MIN_TEXT_LEN or value.startswith(("http://", "https://")):
+            continue
+        seen: set[str] = set()
+        for r in analyzer.analyze(value, language="en", entities=entities, score_threshold=SCORE_THRESHOLD):
+            text = value[r.start:r.end].strip()
+            if PROPER_NAME_RE.match(text) and text not in seen and text not in allow:
+                seen.add(text)
+                kind = "person" if r.entity_type == "PERSON" else "organisation"
+                findings.append(Finding(WARN, CHECK, path, pointer, f"looks like a real {kind} name: {text!r}"))
+    return findings
+
+
+def main(argv: list[str] | None = None) -> int:
+    ensure_utf8_stdout()
+    parser = argparse.ArgumentParser(description="NER scan for person and organisation names")
+    parser.add_argument("--dataset", action="append", default=[],
+                        help="Dataset directory (repeatable; default: the datasets that require anonymization)")
+    args = parser.parse_args(argv)
+    try:
+        analyzer = build_analyzer()
+    except (ImportError, OSError) as exc:
+        print(f"NER scan needs its optional dependencies: pip install -r requirements-ner.txt ({exc})")
+        return 2
+    targets = [Path(d).resolve() for d in args.dataset] or [DATASETS_ROOT / name for name in SCOPE]
+    allow = load_allowlist()
+    findings: list[Finding] = []
+    files = 0
+    for dataset in targets:
+        entities = SCOPE.get(dataset.name, ["PERSON"])
+        if not dataset.is_dir():
+            continue
+        for path in iter_data_files(dataset):
+            data, err = load_json(path, CHECK)
+            files += 1
+            findings += [err] if err else scan_data(path, data, entities, analyzer, allow)
+    print(f"Checked {files} file(s)")
+    print_findings(findings, DATASETS_ROOT.parent)
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

data_validation/reference_data/SOURCES.md

@@ -7,11 +7,14 @@ Lookup tables the validators check identifiers against.
 | DSGAI | `dsgai_entries.json` | 1.0 | OWASP GenAI Data Security Risks and Mitigations 2026 (kept identical to `datasets/_shared/dsgai_taxonomy.json`; a test enforces it) |
 | MITRE ATLAS | `mitre_atlas_techniques.csv` | 2026.09 | latest release listed in `https://raw.githubusercontent.com/mitre-atlas/atlas-data/main/dist/manifest.yaml` |
 | OWASP Top 10s | `owasp_top10.csv` | LLM 2025, LLM 2026, Agentic 2026 | LLM: `https://github.com/GenAI-Security-Project/GenAI-LLM-Top10` (2026 published 2026-08-04). Agentic: `https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/` (published 2025-12-09) |
+| SPDX licenses | `spdx_licenses.csv` | 3.29.0 | `https://raw.githubusercontent.com/spdx/license-list-data/main/json/licenses.json` |
+| Language subtags | `language_subtags.csv` | 2026-09-17 | IANA Language Subtag Registry, `https://www.iana.org/assignments/language-subtag-registry/language-subtag-registry` (languages, scripts, regions) |
+| NIST AI 100-2 | `nist_ai_100_2.csv` | E2025 | NIST AI 100-2e2025, *Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations*, Taxonomy Index, https://doi.org/10.6028/NIST.AI.100-2e2025. The 30 `NISTAML.*` identifiers from its Taxonomy Index and section text (NISTAML.016 and .017 appear only in §3.4.1) |
 | CWE | `cwe_ids.csv` | 4.20 | `https://cwe.mitre.org/data/xml/cwec_latest.xml.zip` (weaknesses, categories and views) |
 
 `owasp_top10.csv` is maintained by hand. The 2026 LLM list renumbered the entries (LLM03 was Supply Chain in 2025 and is Excessive Agency in 2026), so IDs always carry the edition year; `equivalent_2025` gives the 2025 ID each 2026 entry continues (LLM08:2026 Hidden Context Exposure broadens LLM07:2025 System Prompt Leakage). OWASP writes the Agentic IDs without a year (ASI01); this repository adds `:2026` so a future edition cannot collide.
 
-Regenerate the ATLAS and CWE tables with `python update_reference_data.py`; it fills in the Version column. Do not edit those two CSVs by hand.
+Regenerate the ATLAS, CWE, SPDX and language tables with `python update_reference_data.py`; it fills in the Version column. Do not edit those CSVs by hand. `nist_ai_100_2.csv` and `owasp_top10.csv` are maintained by hand when a new edition is published.
 
 `datasets/_shared/dsgai_taxonomy.json` is the DSGAI list the validators use. `dsgai_entries.json` is kept because the Turkish prompt-injection validator reads it.