Skip to content

FAQ: reproduction aggregation in axis vocabulary, and the replication-chain target rule #136

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
LukasWallrich wants to merge 1 commit into
base: main
Choose a base branch
from
+6 −0
Open

FAQ: reproduction aggregation in axis vocabulary, and the replication-chain target rule #136

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions output/flora_faq.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,12 @@ Then it will be listed twice in FLoRA, once as a replication and once as a repro

Each row is one pair of references. A replication study can have multiple studies but their results are aggregated (e.g., based on the authors’ judgment, who may report aggregated results or conflicting results, which we consider as mixed).

For reproductions, the same aggregation applies to each outcome axis separately, in that axis’s own vocabulary: conflicting results across several analyses mean *computational issues* on the computational axis, and *robustness challenges* on the robustness axis. There is no *mixed* value for reproductions.

### Which study is the target when a finding has already been replicated by others?

The target is the study the replication actually re-tests — the one closest to it in the chain of replications — not the chain’s original source. For example, a 2024 paper re-testing a 2010 finding as it was measured in a 2018 replication is entered against the 2018 study.

### When is a finding mixed?

We rely on what replication authors say in the abstract or the report, subject to our interpretation of this. If studies are part of a meta-paper, we still rely on the individual reports where available, and otherwise on the main criterion used by the paper authors.
Expand Down