FAQ: reproduction aggregation in axis vocabulary, and the replication-chain target rule

[LukasWallrich]

Two coding rules that the FLoRA extraction pipeline already applies, but which the FAQ either stated only in replication terms or did not state at all. Both surfaced while writing the FLoRA extractor's outcome prompts (forrtproject/flora-extractor#126), where the FAQ is the authoritative source for coding rules.

1. Reproduction aggregation

"What level is the database?" said that several studies aggregate, and that conflicting results are "considered as mixed". That is the replication vocabulary — reproductions are coded on two independent axes (outcome_computation, outcome_robustness) and have no mixed value at all. A coder following the sentence literally has nowhere to put a reproduction whose analyses disagree.

Added: conflicting results aggregate per axis, in that axis's own terms — computational issues on the computational axis, robustness challenges on the robustness axis, and no mixed for reproductions. The wording matches the coded values exactly.

2. The replication-chain target rule

The FAQ had no entry for this. The rule existed only as a fragment in a project doc — "Replication chains: most likely the closest successor study" — whose direction is genuinely ambiguous in isolation: read one way "successor" points forward from the original, when what is meant is the nearest predecessor that the new paper actually re-tests.

Added as its own question, with an example (a 2024 paper re-testing a 2010 finding as measured in a 2018 replication is entered against the 2018 study).

For discussion — the centrality carve-out and robustness checks

@LukasRoeseler — one related point I did not change, because it is a coding-policy call rather than a wording fix.

"What is the outcome based on?" currently says the outcome "refers to whether the central finding that the replication was designed to test could be replicated and may leave out robustness checks and exploratory analyses."

For replications that is clearly right. For reproductions it reads oddly, because robustness is not peripheral there — it is one of the two axes we code: a reproduction whose result collapses under a reasonable alternative specification is coded robustness challenges, which is exactly a robustness check being included in the outcome. As written, the sentence tells a coder to leave out the thing the robustness axis exists to capture.

Options as I see them:

1. Scope the carve-out explicitly to replications ("for replications, the outcome may leave out robustness checks and exploratory analyses"), leaving the reproduction axes to speak for themselves.
2. Leave it, and rely on the reproduction-specific entries elsewhere in the FAQ to override it for readers who get that far.
3. Something else — if robustness checks are meant to be excluded from a reproduction's robustness verdict in some cases (e.g. checks the reproducers invented rather than ones the original claimed), that distinction would be worth stating, and it would also change how the extractor's prompt should read.

Happy to make whichever edit you prefer; I did not want to guess, since it affects how existing entries were coded.

🤖 Generated with Claude Code

[LukasRoeseler]

Thank you! I suggest we go for option one plus explicitly say that for reproductions, robustness is one of the two axes. And I would not use the term axis but dimensions or facets.