Skip to content

GETTING-STARTED.md

Latest commit

 

History

History
470 lines (356 loc) · 19.7 KB

GETTING-STARTED.md

File metadata and controls

470 lines (356 loc) · 19.7 KB
Raw

Getting Started with PyDiffWatch

A step-by-step guide from install to running PyDiffWatch continuously under your own harness. For the project overview and the no-execution security model, see the README; for authoring detection rules, see RULES.md.

Contents

  1. Install
  2. Choose and wire your LLM endpoint
  3. API keys
  4. Structured-output modes
  5. The operating loop
  6. Running on a harness
  7. State, persistence & containment
  8. Alerts
  9. Heuristic-only mode (no LLM)
  10. Troubleshooting
  11. Detection scope on brand-new packages

1. Install

git clone <your-fork-url> pydiffwatch && cd pydiffwatch
python3 -m venv .venv && . .venv/bin/activate
pip install -e .                 # core: stdlib + PyYAML + defusedxml
pip install -e ".[claude]"       # ONLY if you'll use the Anthropic provider (pulls in the anthropic SDK)

Requires Python 3.11+ (the config loader uses the stdlib tomllib). Verify the CLI:

pydiffwatch -c examples/local-qwen.toml --help

2. Choose and wire your LLM endpoint

PyDiffWatch scores every changed release with the rules engine and escalates anything at or above threshold_t to an LLM for a verdict. The reviewer talks to any OpenAI-compatible endpoint or the Anthropic API — pick the one matching the model server you run, copy its example to pydiffwatch.toml, and edit model/base_url.

cp examples/ollama.toml pydiffwatch.toml      # then edit

Local, OpenAI-compatible, no key — llama-swap / vLLM / LM Studio (e.g. vllm serve Qwen/Qwen2.5-Coder-7B-Instruct --port 8000) — examples/local-qwen.toml:

[reviewer]
provider = "openai"
base_url = "http://localhost:8000/v1"
model = "qwen-singleshot"            # the model name your server exposes
structured_output = "json_schema"

Ollama (ollama serve exposes /v1 on 11434; ollama pull qwen2.5-coder) — examples/ollama.toml:

[reviewer]
provider = "openai"
base_url = "http://localhost:11434/v1"
model = "qwen2.5-coder"
structured_output = "json_object"    # Ollama json_schema support varies by model; loose JSON is safe

llama.cpp (llama-server -m model.gguf --port 8080) — examples/llamacpp.toml:

[reviewer]
provider = "openai"
base_url = "http://localhost:8080/v1"
model = "local"
structured_output = "json_schema"    # recent GBNF-backed builds; drop to json_object on older ones

OpenAI — examples/openai.toml:

[reviewer]
provider = "openai"
base_url = "https://api.openai.com/v1"
model = "gpt-4o-mini"
api_key_env = "OPENAI_API_KEY"
structured_output = "json_schema"

then export OPENAI_API_KEY=sk-... (see §3).

OpenRouter / Groq / Together / any hosted OpenAI-compatible gateway — same shape as OpenAI with that gateway's base_url and key variable:

[reviewer]
provider = "openai"
base_url = "https://openrouter.ai/api/v1"
model = "qwen/qwen-2.5-coder-32b-instruct"
api_key_env = "OPENROUTER_API_KEY"
structured_output = "json_object"

DeepSeek / reasoning ("thinking") models — examples/deepseek.toml:

[reviewer]
provider = "openai"
base_url = "https://api.deepseek.com/v1"
model = "deepseek-chat"
api_key_env = "DEEPSEEK_API_KEY"
structured_output = "json_object"   # DeepSeek 400s on strict json_schema — use json_object
max_output_tokens = 32000           # reasoning eats the budget; leave room for thinking + the verdict

# Optional: pass provider-specific knobs verbatim (reserved core fields always win).
# [reviewer.extra_body]
# reasoning = { enabled = false }   # disable thinking to reclaim output budget (key is provider-specific)

Reasoning models spend output tokens on internal thinking, so a small max_output_tokens truncates the JSON verdict; 32000 leaves room for both. The reviewer also asks the model to emit the decision fields first, so even a truncated verdict carries the classification. Set structured_output = "json_object" — the strict json_schema variant is an OpenAI extension DeepSeek doesn't accept.

Anthropic (needs pip install -e ".[claude]") — examples/anthropic.toml:

[reviewer]
provider = "anthropic"
model = "claude-sonnet-4-6"
escalation_model = "claude-opus-4-8"   # optional: escalate low-confidence verdicts to Opus
structured_output = "json_schema"

then export ANTHROPIC_API_KEY=sk-ant-... (see §3).

3. API keys

The rule: the config file holds the env-var NAME; the shell holds the key. No secret ever lands in a file you might commit, so any pydiffwatch.toml is safe to check in.

For every OpenAI-compatible provider, wire a key in two steps:

  1. In [reviewer], set api_key_env to the name of the variable (not the key): 
    api_key_env = "OPENAI_API_KEY"
  2. Export the key in the environment that runs pydiffwatch: 
    export OPENAI_API_KEY="sk-..."

At request time PyDiffWatch reads $OPENAI_API_KEY and sends Authorization: Bearer <key>. The variable name is yours to choose — use a distinct one per provider so multiple configs coexist (OPENAI_API_KEY, OPENROUTER_API_KEY, GROQ_API_KEY, …). If api_key_env is unset, or the variable is empty, no auth header is sent — which is exactly what a local server wants, so just omit api_key_env for llama-swap / vLLM / Ollama / llama.cpp / LM Studio.

Anthropic is the exception. The Anthropic SDK reads ANTHROPIC_API_KEY from the environment itself, so api_key_env is ignored when provider = "anthropic". Just export ANTHROPIC_API_KEY=.... If the key is missing, that run logs a notice and falls back to heuristic-only — it does not crash.

Getting the key to your scheduler (cron/systemd/Docker/CI), not just your login shell, is the part people miss — see §7 for how each harness injects it.

4. Structured-output modes

structured_output controls how strictly the model is made to return machine-readable JSON:

mode meaning use when
json_schema strict, server-enforced schema the server supports it (OpenAI, vLLM, recent llama.cpp, Anthropic) — preferred
json_object "return valid JSON", no schema Ollama and many gateways
none prompt-only; nothing enforces the shape very small models / last resort

Regardless of mode, the parsed verdict is validated client-side against the review schema. A verdict missing a required field or carrying an out-of-range enum is rejected, and the release degrades to a heuristic alert — never a silent pass. Start at json_schema; step down only if the logs show ReviewUnavailable: non-JSON content.

5. The operating loop

First time only — set the starting point so you process new releases, not all of PyPI history:

pydiffwatch -c pydiffwatch.toml seed-now

(You can skip this: a plain run on a fresh database self-seeds the cursor to "now" and processes nothing that tick, then the next tick polls forward. Use run --backfill to process historical releases from genesis instead.)

Each tick — this is what your scheduler runs:

pydiffwatch -c pydiffwatch.toml run

run pulls every release since the last cursor (capped by max_releases_per_run), diffs each against its prior version, scores it with the ruleset, and escalates anything ≥ threshold_t to the reviewer. Clear-malicious verdicts alert immediately; borderline "suspicious" ones queue for your judgement.

Triage the queue:

pydiffwatch -c pydiffwatch.toml pending                      # suspicious releases awaiting a verdict, with diffs
pydiffwatch -c pydiffwatch.toml adjudicate <id> malicious --note "curl|sh in setup.py"
pydiffwatch -c pydiffwatch.toml evidence <id>                # print the stored flagged payload for a release

adjudicate records benign | malicious | suspicious; a non-benign call emits an alert. evidence prints the payload code captured at detection time and stored in the DB, so it survives the package later being pulled from PyPI. To backfill evidence for older flagged rows captured before evidence storage existed:

pydiffwatch -c pydiffwatch.toml capture-evidence                 # all reportable rows missing evidence
pydiffwatch -c pydiffwatch.toml capture-evidence --release-id <id>
pydiffwatch -c pydiffwatch.toml capture-evidence --all           # widen to every fired-rule row (more re-fetches)

6. The dashboard & the watch daemon

Two extras make PyDiffWatch easier to run and easier to act on: a built-in daemon loop and a local HTML dashboard of verdicts with one-click "report to PyPI" links.

The dashboard renders every reviewed release as a card — malicious and suspicious sorted first and highlighted, benign muted — each with a direct PyPI link, and flagged cards carry a "Report malware on PyPI" action so going from "the tool flagged this" to "reported for takedown" is one click. It's a single self-contained HTML file with no JavaScript; every untrusted string (package name, the model's reasoning, cited code) is HTML-escaped, so a package literally named <script>…</script> can't attack the page.

Generate it from whatever the database already holds:

pydiffwatch -c pydiffwatch.toml dashboard                 # writes .diffwatch/dashboard.html
pydiffwatch -c pydiffwatch.toml dashboard --serve         # also serve it on http://127.0.0.1:8787

--serve binds 127.0.0.1 only (localhost) by default and blocks until Ctrl-C. Use --out PATH to choose the file and --port N to change the port.

To view the dashboard from another device on your LAN, bind all interfaces:

pydiffwatch -c pydiffwatch.toml dashboard --serve --host 0.0.0.0   # reachable at http://<this-host-ip>:8787

Exposing it widens your trust boundary. The page serves your verdict data and renders strings derived from untrusted PyPI packages. It's HTML-escaped against XSS and the server is read-only with no control endpoints, but --host 0.0.0.0 makes it reachable by anyone who can reach this host. Only do it on a network you trust, and keep the default 127.0.0.1 otherwise. The same --host flag works on watch.

The watch daemon is the built-in alternative to wiring up cron/systemd (§7): it scans on an interval, refreshes the dashboard after each tick, and — with --serve — serves it the whole time. One command gives you a running monitor plus a live results page:

pydiffwatch -c pydiffwatch.toml seed-now                  # first time only (start "from now")
pydiffwatch -c pydiffwatch.toml watch --serve             # scan every 5 min + live dashboard
# → open http://127.0.0.1:8787/dashboard.html

--interval N sets the seconds between scans (default 300); --out/--port work as above. A failed scan (network blip, endpoint down) is logged and the daemon keeps going; Ctrl-C stops cleanly. The dashboard's status strip shows whether your model endpoint is reachable and how long ago the last scan ran — start your model server (§2) before watch --serve, or reviews fall back to heuristics until it's up.

It is a foreground process — keep the terminal open, or run it under your agent harness, which will run it as a background task and hand you back the dashboard URL. For unattended, machine-level scheduling, prefer the harness patterns in §7.

7. Running on a harness (cron / systemd / Docker / CI)

PyDiffWatch is a plain CLI over a local SQLite DB; "running it" means invoking run on a schedule under whatever runtime you already operate. All four patterns below are equivalent — pick one. Concurrent runs are safe: a second run that overlaps the first sees the lock, prints run already in progress; exiting, and no-ops.

cron

cron runs with a minimal environment, so inject the key via a small wrapper and use absolute paths. Keep the key out of the crontab itself.

# /opt/pydiffwatch/run.sh   (chmod 700)
#!/usr/bin/env bash
set -euo pipefail
cd /opt/pydiffwatch
source ./.env                 # contains: export OPENAI_API_KEY=sk-...   (chmod 600, git-ignored)
exec .venv/bin/pydiffwatch -c pydiffwatch.toml run
*/15 * * * * /opt/pydiffwatch/run.sh >> /opt/pydiffwatch/run.log 2>&1

systemd timer

Better isolation and journald logging. A oneshot service + a timer.

/etc/systemd/system/pydiffwatch.service:

[Unit]
Description=PyDiffWatch one tick

[Service]
Type=oneshot
WorkingDirectory=/opt/pydiffwatch
EnvironmentFile=/opt/pydiffwatch/.env          # KEY=VALUE lines (NO `export`), e.g. OPENAI_API_KEY=sk-...
ExecStart=/opt/pydiffwatch/.venv/bin/pydiffwatch -c pydiffwatch.toml run

/etc/systemd/system/pydiffwatch.timer:

[Unit]
Description=Run PyDiffWatch every 15 minutes

[Timer]
OnBootSec=2min
OnUnitActiveSec=15min
Persistent=true

[Install]
WantedBy=timers.target
systemctl enable --now pydiffwatch.timer

EnvironmentFile lines are KEY=VALUE, not export KEY=VALUE (that's the cron wrapper's .env).

Docker

Bind-mount the state directory so the cursor and history persist across container runs.

FROM python:3.12-slim
WORKDIR /app
COPY . .
RUN pip install -e .
ENTRYPOINT ["pydiffwatch", "-c", "pydiffwatch.toml"]
docker build -t pydiffwatch .
docker run --rm -v "$PWD/.diffwatch:/app/.diffwatch" -e OPENAI_API_KEY pydiffwatch seed-now
docker run --rm -v "$PWD/.diffwatch:/app/.diffwatch" -e OPENAI_API_KEY pydiffwatch run

Schedule the run line from the host (cron/systemd calling docker run). For a local model, point base_url at the host — http://host.docker.internal:11434/v1 — or share a Docker network with the model container.

CI (GitHub Actions cron)

Works, with two caveats: the DB must persist between runs, and there's no local GPU so the reviewer must be a hosted endpoint. Store the key as an Actions secret.

on:
  schedule:
    - cron: "*/30 * * * *"
jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install -e .
      - uses: actions/cache@v4
        with: { path: .diffwatch, key: pydiffwatch-state }
      - run: pydiffwatch -c pydiffwatch.toml run
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

The Actions cache is best-effort, not durable storage. For anything you rely on, run PyDiffWatch on a host you control and keep .diffwatch/ on a real volume.

8. State, persistence & containment

All state lives under .diffwatch/ (paths configurable via db_path, cache_dir, lock_path):

  • diffwatch.sqlite — the cursor, every processed release, verdicts, alerts, and stored payload evidence.
  • artifact_cache/ — downloaded sdists (size-capped, read in memory, never installed).
  • diffwatch.lock — an exclusive flock that prevents overlapping runs.

Persist .diffwatch/ and you can move PyDiffWatch between machines without losing the cursor or history. The download/extraction caps (max_download_bytes, max_member_bytes, max_total_bytes, max_decompressed_bytes, …) bound how much of any sdist is ever read into memory; the package is never installed, built, imported, or executed — see the README invariant.

Hardening (defense-in-depth). PyDiffWatch installs a process-wide default-deny egress allowlist (pydiffwatch/egress.py) so it can only contact PyPI, the configured reviewer endpoint, and an optional webhook. For production deployments, two guides under docs/hardening/ cover the authoritative OS-level boundaries: egress-allowlist.md (domain-aware proxy / systemd IP allowlist / nftables) and parse-sandbox.md (running the byte-parsing stage under a container/gVisor or a no-network sandboxed subprocess).

9. Alerts

Set webhook_url (top-level, not under [reviewer]) to receive each new alert as a JSON POST — {"text": "..."}, Slack-incoming-webhook compatible:

webhook_url = "https://hooks.slack.com/services/XXX/YYY/ZZZ"

Alerts are also printed to stdout and recorded (deduped) in the DB, so a webhook failure never loses one.

10. Heuristic-only mode (no LLM)

To run with no model at all — rules and weights only, no endpoint required — set:

reviewer_enabled = false

Every release crossing threshold_t becomes a heuristic alert. Useful for a first pass on a box with no GPU and no API budget, or to keep monitoring when your endpoint is down.

11. Troubleshooting

Symptom Cause / fix
ReviewUnavailable: non-JSON content in logs the model isn't honoring the JSON contract. Lower structured_output (json_schema → json_object → none) or use a more capable model. The release still alerted heuristically — nothing was dropped.
Every Anthropic run logs heuristic-only this run ANTHROPIC_API_KEY isn't in the environment the scheduler uses. Put it in the systemd EnvironmentFile / cron wrapper / Actions secret, not just your interactive shell.
401/403 from a hosted endpoint api_key_env names a variable that's unset, empty, or wrong. Check it from the harness's environment: echo $OPENAI_API_KEY.
400/ReviewUnavailable: HTTP Error 400 from DeepSeek (or another reasoning model) the endpoint rejects strict json_schema (an OpenAI-only extension). Set structured_output = "json_object". See examples/deepseek.toml.
DeepSeek verdicts arrive with empty reasoning/cited_hunk or attack_type: none the response truncated — reasoning ate the output budget. Raise max_output_tokens (try 32000) and/or disable thinking via [reviewer.extra_body]. The classification still survives (emitted first).
First run returns processed 0 releases expected — a fresh DB seeds the cursor to "now" and processes nothing that tick; the next tick polls forward. Use run --backfill to process history instead.
a scan is already running … another run holds the lock — the message names the holder pid and the lock file. If it's your scheduled tick, harmless; space the schedule. If nothing is actually running, a prior run hung or was killed mid-fetch and still holds the lock: kill the reported pid and re-run. The lock is an OS advisory lock that frees when its process exits — deleting the lock file does not release a live lock.
Local endpoint refused / connection error the model server isn't up, or base_url is wrong (check the port and the trailing /v1). From Docker, use host.docker.internal, not localhost.

12. Detection scope on brand-new packages

The pipeline's core signal is the version-to-version diff, so a package's first-ever release has no prior version to diff against. new_package_policy controls how those are handled:

Value First-release behavior
surface (default) Scan only the files PyPI auto-runs at install/import — setup.py, setup.cfg, pyproject.toml, __init__.py, conftest.py, sitecustomize.py, .pth — treating each as fully added.
full Scan every .py file in the new package as added (complete coverage, higher volume/noise).
skip Ignore new packages entirely.

Under the default, malware that lives in a non-auto-exec module of a brand-new package (e.g. src/pkg/utils/helper.py) is not scanned — first-release ≠ full scan. Set new_package_policy = "full" if you want complete coverage of first releases and can absorb the extra volume.