Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 60 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,70 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

## [0.1.0] - 2026-10-03

### Changed

- Established the first internally consistent CTRF specification release: the
specification header, inline examples, standalone examples, and conformance
fixtures now identify version `0.1.0`.
- Defined the versioning policy: before `1.0.0`, PATCH releases preserve
compatibility while MINOR releases may contain additions or breaking contract
changes; from `1.0.0`, breaking changes require a MAJOR release.
- No report-shape or validation changes were introduced relative to `v0.0.4`;
producers should emit `"specVersion": "0.1.0"` for this release.

## [0.0.4] - 2026-08-15

> Retrospectively assigned specification snapshot, tagged on 2026-10-03. Files
> at this revision retain their original version metadata.

### Changed

- Clarified `retryAttempts` as the ordered history preceding the final attempt, aligned `retries` counting semantics, and updated the schema and examples accordingly ([#62](https://github.com/ctrf-io/ctrf/pull/62)).
- Clarified `retryAttempts` as the ordered history preceding the final attempt,
aligned `retries` counting semantics, and updated the schema and examples
accordingly ([#62](https://github.com/ctrf-io/ctrf/pull/62)).

## [0.0.3] - 2026-07-26

> Retrospectively assigned specification snapshot, tagged on 2026-10-03. Files
> at this revision retain their original version metadata.

### Added

- Added an optional identity model for CTRF documents, logical runs, test cases, executions, attempts, attachments, and shards ([#57](https://github.com/ctrf-io/ctrf/pull/57)).

### Changed

- Clarified namespace guidance for `extra` extension keys and examples ([#56](https://github.com/ctrf-io/ctrf/pull/56)).
- Clarified immutability guidance for emitted CTRF report artifacts ([#55](https://github.com/ctrf-io/ctrf/pull/55)).
- Clarified `tags` as simple keyless classifications and `labels` as structured key-value test metadata ([#54](https://github.com/ctrf-io/ctrf/pull/54)).
- Allowed non-empty arrays of strings, numbers, and booleans as label values ([#54](https://github.com/ctrf-io/ctrf/pull/54)).

## [0.0.2] - 2026-02-07

> Retrospectively assigned specification snapshot, tagged on 2026-10-03. Files
> at this revision retain their original version metadata.

### Added

- Added structured scalar `labels` metadata to test objects
([#51](https://github.com/ctrf-io/ctrf/pull/51)).

## [0.0.1] - 2026-01-24

> Retrospectively assigned specification snapshot, tagged on 2026-10-03. Files
> at this revision retain their original version metadata.

### Added

- Established the standalone CTRF specification, normative JSON Schema,
examples, and conformance tests
([#50](https://github.com/ctrf-io/ctrf/pull/50)).

[Unreleased]: https://github.com/ctrf-io/ctrf/compare/v0.1.0...HEAD
[0.1.0]: https://github.com/ctrf-io/ctrf/compare/v0.0.4...v0.1.0
[0.0.4]: https://github.com/ctrf-io/ctrf/compare/v0.0.3...v0.0.4
[0.0.3]: https://github.com/ctrf-io/ctrf/compare/v0.0.2...v0.0.3
[0.0.2]: https://github.com/ctrf-io/ctrf/compare/v0.0.1...v0.0.2
[0.0.1]: https://github.com/ctrf-io/ctrf/tree/v0.0.1
11 changes: 8 additions & 3 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,8 @@ The normative JSON Schema lives at:
Schema changes MUST:

- match the written specification
- avoid breaking existing valid documents
- avoid breaking existing valid documents except for an explicitly approved
pre-1.0 MINOR release
- use consistent validation constraints

If the schema and specification disagree, the **written specification takes precedence**.
Expand All @@ -88,8 +89,12 @@ If the schema and specification disagree, the **written specification takes prec

Backward compatibility is a core CTRF principle.

- PATCH and MINOR releases MUST NOT introduce breaking changes
- Breaking changes MUST be explicit and well-justified
- PATCH releases MUST NOT introduce breaking changes
- Before `1.0.0`, MINOR releases MAY introduce breaking contract changes
- Beginning with `1.0.0`, breaking changes require a MAJOR release and MINOR
releases contain only backward-compatible additions
- Breaking changes MUST be explicit, well-justified, and documented with
migration guidance when action is required

---

Expand Down
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,8 @@ The written specification defines the semantics and rules.

CTRF follows Semantic Versioning.

Releases are defined in [Releases](https://github.com/ctrf-io/ctrf/releases).
Published specification versions are defined by annotated Git tags. See the
[release process](RELEASE.md) for the current pre-1.0 policy.

## Reference Implementation

Expand Down
70 changes: 70 additions & 0 deletions RELEASE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
# CTRF Releases

CTRF is currently in pre-1.0 development.

## Versioning

CTRF specification versions and the `specVersion` field use
`MAJOR.MINOR.PATCH` version numbers.

Before `1.0.0`:

- MINOR versions may add capabilities or introduce breaking contract changes.
- PATCH versions preserve compatibility and contain compatible corrections or
clarifications.
- Breaking changes are identified in `CHANGELOG.md`, with migration guidance
when producer or consumer action is required.

Different pre-1.0 MINOR versions may be incompatible. PATCH versions within the
same MINOR version are compatible.

Beginning with `1.0.0`:

- MAJOR versions contain breaking changes.
- MINOR versions contain backward-compatible additions.
- PATCH versions contain backward-compatible corrections and clarifications.

## Releases

Before `1.0.0`, CTRF specification releases are published as annotated Git tags
named `vMAJOR.MINOR.PATCH`.

The tag and its repository contents are the canonical release record. Published
tags are immutable; corrections are published as a new version.

GitHub Releases are not used during the pre-1.0 tags-only period.

Release changes are reviewed through a pull request. Merging the pull request
does not publish the release; publication occurs when the version tag is pushed.

## Release contents

The specification, JSON Schema, examples, conformance fixtures, and changelog
are versioned together.

For each release:

- the specification header identifies the release version and date;
- inline examples, standalone examples, and valid conformance fixtures use the
same `specVersion`;
- `CHANGELOG.md` describes the changes from the preceding version; and
- the schema embedded in `spec/ctrf.md` matches
`schema/ctrf.schema.json` byte-for-byte.

Schema formatting and linting, example validation, and the reference and
normative conformance suites form the release verification record.

If the written specification and JSON Schema disagree, the written
specification takes precedence and the inconsistency is corrected in a new
release.

## Current release history

The `v0.0.1` through `v0.0.4` tags were assigned retrospectively to historical
specification snapshots. Files at those commits retain their original version
metadata and may not identify the version shown by the tag.

Version `0.1.0` is the first release in which the specification header, inline
examples, standalone examples, and valid conformance fixtures identify the same
specification version. It introduces no report-shape or validation change from
`v0.0.4`; producers adopting it use `"specVersion": "0.1.0"`.
2 changes: 1 addition & 1 deletion examples/comprehensive.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"reportFormat": "CTRF",
"specVersion": "1.0.0",
"specVersion": "0.1.0",
"reportId": "9d2c6a10-3f7a-4e22-9a8f-1a2b3c4d5e6f",
"runId": "run-2025-11-24-pr-123",
"timestamp": "2025-11-24T12:00:00Z",
Expand Down
2 changes: 1 addition & 1 deletion examples/minimal.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"reportFormat": "CTRF",
"specVersion": "1.0.0",
"specVersion": "0.1.0",
"results": {
"tool": {
"name": "example-runner"
Expand Down
2 changes: 1 addition & 1 deletion examples/with-diagnostics.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"reportFormat": "CTRF",
"specVersion": "1.0.0",
"specVersion": "0.1.0",
"results": {
"tool": {
"name": "example-runner",
Expand Down
2 changes: 1 addition & 1 deletion examples/with-insights.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"reportFormat": "CTRF",
"specVersion": "1.0.0",
"specVersion": "0.1.0",
"reportId": "7c4e1c20-7c89-4f30-9b52-1f6f9d6b8f21",
"results": {
"tool": {
Expand Down
2 changes: 1 addition & 1 deletion examples/with-retries.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"reportFormat": "CTRF",
"specVersion": "1.0.0",
"specVersion": "0.1.0",
"results": {
"tool": {
"name": "example-runner",
Expand Down
51 changes: 37 additions & 14 deletions spec/ctrf.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,10 @@

## Common Test Report Format

**Version:** 0.0.0
**Version:** 0.1.0
(This version corresponds directly to the CTRF `specVersion` field.)

**Date:** 2025-11-24
**Date:** 2026-10-03

**Status:** Working Draft

Expand Down Expand Up @@ -206,14 +206,21 @@ This reduces ambiguity and ensures consistent interpretation across consumers.

CTRF follows Semantic Versioning and is designed to evolve without breaking existing consumers.

Backward-compatible additions are introduced through:
Before CTRF `1.0.0`, MINOR versions may introduce additions or breaking contract
changes, while PATCH versions preserve compatibility.

Beginning with CTRF `1.0.0`:

- MAJOR versions contain breaking changes.
- MINOR versions contain backward-compatible additions.
- PATCH versions contain backward-compatible corrections and clarifications.

Backward-compatible additions may be introduced through:

- optional fields
- new insight metrics
- use of `extra` objects

Breaking changes are reserved for major version increments.

---

### 2.9. Interoperability Over Optimization
Expand Down Expand Up @@ -1964,9 +1971,25 @@ It is the ONLY permitted extension point within the baseline object.

CTRF follows **Semantic Versioning**.

- MAJOR = breaking changes
- MINOR = backward-compatible additions
- PATCH = non-breaking fixes
This document defines CTRF version `0.1.0`.

Before CTRF `1.0.0`:

- MINOR versions MAY add capabilities or introduce breaking contract changes.
- PATCH versions MUST preserve compatibility and are reserved for compatible
corrections and clarifications.
- Breaking changes MUST be identified in the changelog and include migration
guidance where action is required from producers or consumers.

Consumers MUST treat different pre-1.0 MINOR versions as potentially
incompatible. Consumers SHOULD support PATCH releases within a supported
pre-1.0 MINOR version.

Beginning with CTRF `1.0.0`:

- MAJOR versions contain breaking changes.
- MINOR versions contain backward-compatible additions.
- PATCH versions contain backward-compatible corrections and clarifications.

Consumers MUST reject incompatible MAJOR versions.

Expand Down Expand Up @@ -2967,7 +2990,7 @@ It includes:
```json title="Minimal CTRF document"
{
"reportFormat": "CTRF",
"specVersion": "0.0.0",
"specVersion": "0.1.0",
"results": {
"tool": {
"name": "example-runner"
Expand Down Expand Up @@ -3008,7 +3031,7 @@ It includes:
```json title="CTRF document with retries"
{
"reportFormat": "CTRF",
"specVersion": "0.0.0",
"specVersion": "0.1.0",
"results": {
"tool": {
"name": "example-runner",
Expand Down Expand Up @@ -3065,7 +3088,7 @@ It includes:
```json title="CTRF document with diagnostics"
{
"reportFormat": "CTRF",
"specVersion": "0.0.0",
"specVersion": "0.1.0",
"results": {
"tool": {
"name": "example-runner",
Expand Down Expand Up @@ -3129,7 +3152,7 @@ It includes:
```json title="CTRF document with insights and baseline"
{
"reportFormat": "CTRF",
"specVersion": "0.0.0",
"specVersion": "0.1.0",
"reportId": "7c4e1c20-7c89-4f30-9b52-1f6f9d6b8f21",
"results": {
"tool": {
Expand Down Expand Up @@ -3232,7 +3255,7 @@ It includes:
```json title="Comprehensive CTRF document"
{
"reportFormat": "CTRF",
"specVersion": "0.0.0",
"specVersion": "0.1.0",
"reportId": "9d2c6a10-3f7a-4e22-9a8f-1a2b3c4d5e6f",
"runId": "run-20251124-e2e-staging",
"timestamp": "2025-11-24T12:00:00Z",
Expand Down Expand Up @@ -3426,7 +3449,7 @@ The `ctrf.` and `ctrf/` namespace prefixes are reserved for CTRF-defined extensi
```json title="CTRF document with namespaced extra objects"
{
"reportFormat": "CTRF",
"specVersion": "0.0.0",
"specVersion": "0.1.0",
"extra": {
"myorg.ci": {
"pipeline": {
Expand Down
Loading
Loading