Add scanner query api

[wagnerand-moz]

Fixes #16332.

Description

This patch adds a REST API for ScannerQueryRule and ScannerQueryResult, so scanner query rules can be created, run, aborted, and iterated on programmatically instead of only through the Django admin.

New endpoints (JWT API-key auth; reads gated by ADMIN_SCANNERS_QUERY_VIEW, writes/run/abort by ADMIN_SCANNERS_QUERY_EDIT):

• GET/POST /api/v5/scanner/query-rules/: list / create a query rule.
• GET/PATCH/PUT/DELETE /api/v5/scanner/query-rules/{id}/: read / edit / delete.
  • Editing is only allowed while the rule is new.
  • name and scanner are frozen after creation.
  • For yara rules the name is inferred from the definition (matching the admin) and required for narc.
• POST /api/v5/scanner/query-rules/{id}/run/: schedule a run (enqueues run_scanner_query_rule).
• POST /api/v5/scanner/query-rules/{id}/abort/: abort a run.
• GET /api/v5/scanner/query-rules/{id}/results/: the rule's matches (nested read-only viewset).
  • Supports filtering (was_blocked, was_promoted, was_signed, addon_disabled_by_user, channel, addon_status, file_status),
  • and ordering (?sort= one of id, addon_adu, version_created, prefix - for descending).
  • Each result exposes a minimal nested version (with its addon guid/average_daily_users).

Context

Scanner query rules (ad-hoc YARA/NARC rules run across the whole add-on corpus) can currently only be authored and run via django admin. This exposes the same lifecycle over HTTP so that an iteration loop can be automated.

Testing

1. Grant a user the Admin:ScannersQueryView and Admin:ScannersQueryEdit groups and get a JWT API key for them.
2. POST /api/v5/scanner/query-rules/ with {"scanner": "yara", "definition": "rule my_rule { condition: true }"} --> expect 201. The response has scanner: "yara", state: "new", and name inferred as "my_rule" from the definition (for narc, name is required).
   a) Posting an uncompilable definition --> expect 400 with a definition error.
3. POST /api/v5/scanner/query-rules/{id}/run/ --> expect 202. The rule state becomes "scheduled", and a run_scanner_query_rule task is enqueued.
   a) Running a non-new rule --> expect 409.
4. Poll GET /api/v5/scanner/query-rules/{id}/ and watch state/completion_rate/completed.
   a) Once complete, GET .../{id}/results/ returns the matches; try filtering/sorting, e.g. ?was_blocked=true, ?channel=listed, ?sort=-addon_adu.
5. PATCH a new rule --> expect 200.
   a) PATCH a non-new rule --> expect 400.
   b) Requests without a valid JWT key --> expect 401.
   c) Requests with a key lacking permissions --> expect 403.

Checklist

• Add #ISSUENUM at the top of your PR to an existing open issue in the mozilla/addons repository.
• Successfully verified the change locally.
• The change is covered by automated tests, or otherwise indicated why doing so is unnecessary/impossible.
• Add or update relevant docs reflecting the changes made.

[eviljeff]

it's difficult to judge without knowing the problem it's trying to solve (which would be in the issue...)

[wagnerand-moz]

it's difficult to judge without knowing the problem it's trying to solve (which would be in the issue...)

It's in the Context. I'd like to programmatically/automatically create scanner query rules, assess their results (i.e. matches) and use them to refine the rules to increase accuracy and reduce false positives, so that at the end, I can add that as a scanner rule.

[diox]

This needs docs, and conflicts should be fixed.

[wagnerand-moz]

@diox thanks. What kind of docs are you referring to?

[diox]

API docs (https://github.com/mozilla/addons-server/tree/master/docs/topics/api) - even internal APIs deserve documentation.

[wagnerand-moz]

Merge conflicts resolved, docs added.

[diox]

You have no tests for configuration, is it working ? Specifically:

• Default configuration should be set after creation
• Modifying configuration should work
• Invalid configuration (not obeying schema) should raise an error

We should also have docs about what can be done through the configuration.

[wagnerand-moz]

• Addressed the review comments
• Updated the description to reflect current state
• Manually tested the API using the attached script test_scanner_api.py

[diox]

Just in case, can you add tests covering patch/put/post/delete without permissions as well (including the abort/run actions) ?

[wagnerand-moz]

Just in case, can you add tests covering patch/put/post/delete without permissions as well (including the abort/run actions) ?

Hah, I had considered that! Will add them.

[wagnerand-moz]

Test updated.