Test Suite Compliance

Repository for collecting and displaying Array API test suite compliance data.

The Array API test suite is the official test suite for measuring compliance to the Python Array API standard. This repository is intended to provide a global vantage point for evaluating Array API compliance across the Scientific Python Ecosystem and to make identifying (and hopefully fixing) interoperability issues as easy as possible.

The test suite compliance dashboard is comprised of two parts:

1. Harvester: collects test suite reports from registered array libraries which seek to implement the standard.
2. Dashboard: public website for displaying test suite results.

---

Quick Start

If you are an array library maintainer wanting to report Array API test suite results, you need to do three things:

1. Set up a public endpoint from which your test suite results can be regularly retrieved. This can be a GitHub workflow artifact, a public URL, or some other supported endpoint mechanism.
2. Register your library's public endpoint(s) with this repository. See Registration below.
3. Once your library is registered, check back within 24-48 hours to confirm that the dashboard is displaying test suite results for your library.

For array libraries already using GitHub workflows for running their test suites (including against the Array API test suite), see the demonstration GitHub workflow for how to run the test suite and generate a valid test suite report JSON.

Note

The demonstration GitHub workflow is configured to test the array_api_strict library on a cron schedule; however, using a cron schedule is not a requirement for test suite reporting. You are free to run the test suite on whatever schedule and/or triggers make sense for your library (e.g., on push, etc). We do ask, however, that you only report test suite results for tests running against your main branch, thus reflecting the code that your users will actually run.

---

Registration

If you are the maintainer of an array library adopting the Array API standard and would like to submit test suite results, you're in the right place!

To add your library, please do the following:

1. Fork this repository.

2. Edit registry/data.json.

3. Add a new entry to the libraries field and include the following data:

   • id: unique identifier (e.g., numpy, array-api-strict, etc). This is an immutable registry identifier. Changing an identifier creates a new history, so renames should be deliberate migrations.

   • name: human-readable display name (e.g., NumPy, Array API Strict, etc).

   • report_name: exact expected report name field value. This field is intended for projects whose display or registry name differs from their installed distribution name. If not provided, defaults to id. (optional)

   • sources: list one or more retrieval endpoints for harvesting test suite reports. All source objects must have the following fields:

     • id: stable identity for the retrieval endpoint. An endpoint may publish one or more reports.
     • type: the type of endpoint. Must be either: 'url' or 'ci_artifact'.

     If type is url, then a source object must have the following field:

     • url: URL for the published artifact(s) (e.g., 'https://example.org/array-api-compliance/latest.json'). Non-HTTPS URLs are not supported. Credentials should not be embedded in URLs.

     If type is ci_artifact, then a source object must have the following fields:

     • provider: name of the CI provider hosting the artifact. Currently, only 'github' is supported.

     • project: provider-native human locator. For GitHub, this is owner/repo, not a URL.

     • ref: structured reference for resolving an artifact. If provider is github, this should be an object with the following fields:

       • kind: structured reference "kind". This should be equal to 'branch'.
       • name: branch name (e.g., 'main').

     • selector: provider-specific artifact. If provider is github, this should be an object with the following fields:

       • workflow: workflow filename.
       • artifact: artifact name.

     Both url and ci_artifact sources have the following optional fields:

     • file: specific file member within a published ZIP archive to use as a test suite result. Other files within the ZIP archive are ignored. (optional)

4. Set up your library to begin publishing test suite results. See below concerning Test Suite Artifacts.

5. Open a PR against this repository which adds the registration entry. Once merged, automated workflows will begin monitoring the registration endpoint(s) and pulling the latest published artifact on a daily basis.

6. Check back on the dashboard within 24-48 hours. If all went according to plan, your library should be displayed in the dashboard alongside other registered array libraries.

Note

There may be a 24-48 hour lag between when an array library registers with this repository and when test suite results start appearing in the dashboard. This is expected and stems from the fact that this repository polls registered array library endpoints once every 24 hours.

Examples

GitHub

The following is an example registration entry for a GitHub workflow artifact:

{
  "id": "array-api-strict",
  "name": "Array API Strict",
  "report_name": "array-api-strict",
  "sources": [
    {
      "id": "demo",
      "type": "ci_artifact",
      "provider": "github",
      "project": "data-apis/array-api-tests-compliance-data",
      "ref": {
        "kind": "branch",
        "name": "main"
      },
      "selector": {
        "workflow": "demo.yml",
        "artifact": "array_api_compliance.json"
      }
    }
  ]
}

URL

{
  "id": "array-api-strict",
  "name": "Array API Strict",
  "report_name": "array-api-strict",
  "sources": [
    {
      "id": "stable-latest",
      "type": "url",
      "url": "https://example.org/array-api-compliance/latest.json"
    }
  ]
}

---

Test Suite Artifacts

Harvest Endpoints

GitHub Workflows

To publish test suite results using GitHub workflows, consult the demonstration workflow file present in this repository.

URLs

When publishing test suite results to a public URL, you should always publish to the same static URL (e.g., latest.json), as it will be that single endpoint which is queried for the latest test suite results.

Upon harvesting, automated workflows in this repository will determine whether the downloaded artifact has already been harvested after extraction, normalization, and a computed checksum.

JSON Reports

The basis of all test suite reporting artifacts is the JSON report generated by pytest when running the Array API test suite.

cd array-api-tests && \
  ARRAY_API_TESTS_MODULE="array_api_strict" \
  ARRAY_API_TESTS_VERSION="2025.12" \
  python -m pytest array_api_tests/ \
    --json-report

See the test suite for further information concerning test suite configuration.

Once the raw report is generated, you need to augment the generated JSON with additional metadata having the following fields:

Tip

See the demonstration workflow for an example of including metadata with the raw report.

• schema: report schema version (e.g., v1).

• name: array library name. This must match the registration report_name, as discussed above for registering an array library (see Registration).

• version: array library version. This is the version of your library against which the test suite ran.

• api_version: Array API version which your library was tested against.

• timestamp: UTC timestamp indicating when the report was generated (e.g., 2026-08-24T10:34:27Z).

• platform: an object containing the following fields:

  • description: platform description as returned by platform.platform() (e.g., Linux-6.17.0-1022-azure-x86_64-with-glibc2.39).
  • release: platform release version as returned by platform.release() (e.g., 6.17.0-1022-azure).
  • machine: machine type as returned by platform.machine() (e.g., x86_64).
  • system: operating system as returned by platform.system() (e.g., Linux).

• execution_target: an object containing the following fields:

  • kind: execution target kind. Must be one of the following: 'cpu', 'gpu', 'tpu', or 'other'.

  If kind is not cpu, the following additional field is required:

  • backend: backend name (e.g., cuda, rocm, mps, oneapi, xla, opencl, etc). Must be lowercase.

  The following additional fields are optional:

  • device_model: device model when the device is known and may affect support or behavior (e.g., NVIDIA H100 80GB HBM3, Apple M4 Max GPU, Intel Xeon Platinum 8481C, etc).
  • runtime_version: runtime version when the user-space execution runtime has a meaningful version (e.g., CUDA runtime 12.8, ROCm 6.3.1, oneAPI 2026.0, etc).
  • driver_version: driver version when a separate host/device driver is identifiable and relevant (e.g., NVIDIA driver 570.124.06, Intel Level Zero driver 1.6.32961, etc).

• python: Python version as returned by platform.python_version() (e.g., 3.14.7).

• test_suite: test suite revision as an exact 40-character hexadecimal commit SHA.

• data: test suite result object containing the following fields, all of which can be obtained from the raw test suite report JSON:

  • duration: test suite execution duration.
  • exitcode: test suite exit code. Must be either 0 or 1.
  • summary: test suite summary, including the number of collected, failed, skipped, and passing tests, along with the total.
  • tests: array containing individual test suite results. The number of individual tests must agree with the test suite summary.

Example

The following is an example test suite report:

{
  "schema": "v1",
  "name": "array-api-strict",
  "version": "2.6.1",
  "api_version": "2025.12",
  "timestamp": "2026-08-24T10:34:27Z",
  "platform": {
    "description": "Linux-6.17.0-1022-azure-x86_64-with-glibc2.39",
    "system": "Linux",
    "release": "6.17.0-1022-azure",
    "machine": "x86_64"
  },
  "execution_target": {
    "kind": "cpu"
  },
  "python": "3.14.7",
  "test_suite": "05b12afc0811fdac80028bc0454d6ca0aefb8101",
  "data": {
    "duration": 61.62217926979065,
    "exitcode": 1,
    "summary": {
      "passed": 1353,
      "skipped": 5,
      "failed": 24,
      "total": 1382,
      "collected": 1382
    },
    "tests": [
      {...},
      {...},
      ...
    ]
  }
}

Execution Targets

The following are example execution_target values.

CUDA

{
  "kind": "gpu",
  "backend": "cuda",
  "device_model": "NVIDIA H100 80GB HBM3",
  "runtime_version": "12.8",
  "driver_version": "570.124.06"
}

ROCm

{
  "kind": "gpu",
  "backend": "rocm",
  "device_model": "AMD Instinct MI300X",
  "runtime_version": "6.3.1"
}

MPS

{
  "kind": "gpu",
  "backend": "mps",
  "device_model": "Apple M4 Max GPU"
}

CPU

Typically, specifying kind as cpu is enough for native CPU runs; however, there may be times when additional information is useful, such as when a specific execution stack is selected, when tests run on dedicated hardware whose model is known and relevant, and when the runtime version may explain a change in behavior. For example,

{
  "kind": "cpu",
  "backend": "oneapi",
  "device_model": "Intel Xeon Platinum 8481C",
  "runtime_version": "2026.0"
}

---

Notes

• Artifacts may be published as an individual test suite report JSON file or a ZIP file containing one or more JSON files.
• External repositories are not required to install a GitHub App or provide credentials. GitHub workflow artifacts must be published on public GitHub repositories.
• For authoritative registry and compliance schemas, see the schemas directory.
• Test suite results are stored in the data directory as compressed records. data/index.json provides a record index, along with summarized test suite results.
• Harvesting uses kind, backend, and device_model as stable series dimensions. Exact Python and platform details and the complete execution target, including runtime and driver versions, identify a matrix variant within that series. Thus, reports for multiple runtime or driver versions can coexist in one artifact while remaining part of the same broader trend.
• Harvesting test suite reports happens daily and only the latest artifact is resolved. Any artifacts published between harvest times are ignored.

---

Contact

Encountered a bug? Please file a GitHub issue. Cheers!

For other discussion, please see the Array API repository and feel free to join one of the regular workgroup meetings.

---

License

See LICENSE.

Copyright

Copyright © 2026. Python Data APIs Consortium.