reference-comparison.md

Compare reference and synthetic OHLCV

Compare two CSV files from the installed package

Use compare when you already have the reference bars and a synthetic series. Supply their common candle interval explicitly, in whole seconds:

pricegenerator compare reference.csv synthetic.csv --timeframe-seconds 3600 --tolerances tolerances.json --output comparison.json

Both files must follow the canonical CSV contract with explicit timezone offsets. Each series needs at least two candles and exactly one interval between adjacent candles. Choose contiguous windows, including when working with market-session data. Use the same volume units and a comparable sampling and adjustment policy for both series. The two windows are analyzed independently and can have different timestamps and row counts.

The optional tolerances.json must be an object of finite, nonnegative numbers keyed by comparison metric. For example, declare an absolute log-return volatility tolerance before opening the evaluation window:

{"log_return_std": 0.005}

The JSON report includes schema_version, metric_version, the declared interval, both canonical data fingerprints, row counts, separate analyses and the result for each specified tolerance. An undefined statistic or an omitted tolerance leaves that metric's within_tolerance at null. A valid report exits with status zero even if a metric exceeds its tolerance. Invalid input exits with status two before writing a report. Omit --output to print the same report to standard output.

Compare a historical window with one generated scenario

python -m tools.compare_market evaluates a caller-selected historical-data window against one independently generated scenario using the declared model. It uses the modern canonical reader, Generate, Compare and scenario manifest APIs. No historical runtime, indicator backend, fitting, seed search, market-calendar inference, gap filling, price adjustment or volume normalization is performed.

A canonical headered UTC OHLCV file is the default:

python -m tools.compare_market reference.csv --config config.json --output comparison.json --markdown comparison.md

config.json is a GenerationConfig JSON object. Omitted fields use library defaults, never estimates from the reference. The complete effective configuration, seed, model/RNG/library identity, environment, input-source hashes and replayable scenario manifest are always recorded. With no config file the complete library defaults apply, including its 150-row horizon. Reference and synthetic row counts may differ and remain explicit.

Use --start 2015-01-01T00:00:00Z --end 2016-01-01T00:00:00Z for an inclusive UTC reference window. Bounds require explicit offsets. The runner records both requested and actual endpoints, source row positions, total/selected counts, raw source-byte SHA-256 and the canonical selected-window fingerprint. It parses exactly the bytes that it hashes. It does not interpolate missing rows.

Optional --tolerances tolerances.json accepts predeclared absolute thresholds for the modern comparison metrics, for example:

{"log_return_std": 0.005, "volume_absolute_return_correlation": 0.1}

Choose thresholds before inspecting the evaluation window. Unlisted metrics receive no decision, and undefined statistics cannot pass. Defaults supply no thresholds and make no acceptance claim. Exit status zero means the comparison executed and reports were written; it is not a global model acceptance result. Invalid schemas, unknown metrics, malformed/duplicate JSON keys, nonfinite values, ambiguous window timestamps and aliased input/output paths fail explicitly. Existing inputs are never overwritten by output aliases.

The JSON contains selected marginal distributions including tails/outlier fractions, all ten return/absolute-return/squared-return ACF lags, activity correlations and drawdown/path summaries. It omits observation-sized vectors. The console and optional Markdown file show the same metric table and limits. Both outputs retain the generating command. Source hashes identify explicitly used modern implementation modules and the recipe, not every file present in the working directory.

Bundled AFLT fixture

The first recorded reference is the existing tests/AFLT_day.csv: 2,775 rows from 2010-01-11 to 2021-01-25. It is a headerless seven-column file containing YYYY.MM.DD,HH:MM[:SS],open,high,low,close,volume. The tool-only --format aflt-day adapter is explicit and requires --timezone. It does not change the canonical CSV API or expose a compatibility runtime. No instrument, venue, timezone, volume-unit or adjusted-price assertion can be verified from this file alone. Europe/Moscow below is a declared assumption, not metadata inferred from its filename. Installed pandas/pytz/tzdata versions are recorded with that assumption; the resulting UTC fingerprint is explicit.

Reproduce the committed machine report and human report:

python -m tools.compare_market tests/AFLT_day.csv --format aflt-day --timezone Europe/Moscow --config docs/evidence/reference-aflt-config.json --output docs/evidence/reference-aflt.json --markdown docs/evidence/reference-aflt.md

The declared configuration fixes seed 20261016, 2,775 observations, one-day synthetic intervals and a UTC start. Other parameters, including initial price100 and volume base1000, remain library defaults; they were not fitted to AFLT. The reference retains trading gaps and historical timezone-offset changes, while the synthetic path uses uniform daily intervals. Consequently the synthetic and reference elapsed calendar horizons differ. Return diagnostics measure successive observations, not returns normalized to a common trading-session or wall-clock duration. Volume units, price levels and adjustment policies are not harmonized.

For this one scenario the reference/synthetic log-return standard deviations are about0.02099/0.02245, while mean volumes are about2,902,683/12,832 and volume/absolute-return correlations are about0.353/0.137. Numerical closeness of one statistic does not validate the remaining distributions or dynamics. No tolerances were supplied: every decision is not assessed. The evidence is an auditable descriptive comparison, not calibration or a realism claim.

Independent holdout protocol

1. Record the question, instrument, venue, sampling cadence, time zone, price adjustments, volume units, missing-bar policy, source version and permitted use. Record the source hash and exact UTC windows and row counts. Publish raw data only when redistribution is permitted.
2. Split observations chronologically. Use the earlier window for fitting parameters, choosing metrics and setting numerical acceptance thresholds. Reserve the later window for evaluation. For rolling metrics, use only inputs available at the observation time and keep each scored window inside the holdout. Declare how much history each metric needs before scoring.
3. Before opening the holdout, record the effective config, model and library versions, seed list, simulated horizon, metrics, acceptance rule and uncertainty method. Match the comparison cadence and effective horizon, or disclose the discrepancy. For a multi-seed claim, report the distribution across the declared runs and retain every run, including failures and undefined metrics. Account for dependence between overlapping observations when quantifying uncertainty.
4. Record which tests passed, failed or could not be assessed, with actual sample sizes, limits and report or manifest hashes. Treat changes made after viewing the evaluation as new hypotheses and reserve a fresh holdout.

For the fuzzy diagnostic, fit the reference volatility and activity levels on the earlier window and freeze the feature definitions, lookback, profile and backend version. To claim that the score detects a regime, declare an independent target and baseline before evaluating. Otherwise report score and clipping distributions as descriptive results.

The AFLT report above has limited source provenance and no acceptance threshold. The Bitstamp chronological holdout documents a separate evaluation with explicit training and holdout windows.

Focused validation:

python -m pytest -q tests/test_compare_market.py
python -m ruff check --select E4,E7,E9,F,D --ignore D202,D203,D213 tools/compare_market.py tests/test_compare_market.py