Calibration test · replication · hard mode

Same test.
No surface clues this time.

The first version was too easy, and the reason was a design flaw: the failed findings were mostly odd priming studies and the survivors were mostly classic judgment effects. You could score well without knowing anything, just by pattern-matching on vibe.

So every pair here is matched — same field, usually the same journal and roughly the same years. Two cooperation experiments from Nature. Two classroom interventions from Science. Two embodiment studies from 2010. One of each pair survived a high-powered replication. One didn't.

Fifteen pairs. ← and → work. Take your time — rushing is detected.

Result · hard mode

Here's how you did.

your fifteen callsgreen = right · amber = wrong

—

Accuracy

How often you picked the finding that actually held up.

—

Fame trap

How often you backed the better-known finding on pairs where the better-known one is the one that collapsed.

The base rates you're being compared against

39%

of 100 psychology studies replicated in the Reproducibility Project: Psychology, with replication effects averaging about half the original size.

62%

of 21 social science experiments from Nature and Science replicated in the Social Sciences Replication Project — 13 of 21, with effects about 50% of the originals.

49%

of 164 papers replicated in SCORE, the seven-year, 865-researcher project published in Nature in April 2026 — with replication effect sizes less than half the originals.

Is this skill even real? The evidence is now split.

Before the Social Sciences Replication Project ran, researchers bet on the outcomes in prediction markets, and the markets called it well — expert forecasters correctly predicted replicability for close to three quarters of the studies. That result is why a test like this seemed worth building.

But SCORE, finishing in 2026 on a far larger and more varied sample, found the opposite: no single indicator of trustworthiness stood out, expert and machine predictions related only modestly to actual replicability, and AI models trained to imitate accurate human forecasters still failed to pick the winners. The one factor that did correlate strongly with reproducibility was mundane — whether the data and code had been made available.

Which means your score above is genuinely ambiguous. On a curated set of famous studies, forecasting works. Across the literature as a whole, it may not. Both things appear to be true, and nobody has fully reconciled them.

Your misses

Back to the easy one

Sources. Pairs are built from the Social Sciences Replication Project (Camerer et al., Nature Human Behaviour, 2018 — 21 studies from Nature and Science, 2010–2015, of which 13 replicated), Many Labs 1 (Klein et al., 36 samples, 6,344 participants), and the registered replication reports on ego depletion (Hagger et al., 23 labs) and facial feedback (Wagenmakers et al., 17 labs). Base rates also cite the Reproducibility Project: Psychology (Open Science Collaboration, 2015) and SCORE (Miske et al., Aczel et al., Tyner et al., Nature 652, 2026).

Caveats. "Replicated" here means a significant effect in the same direction in the replication attempt — the standard indicator, but not the only defensible one, and the SSRP itself reports rates from 57% to 67% depending on which is used. A single failed replication is evidence, not a verdict; several of these effects remain actively debated, and in at least one case (retrieval practice vs. concept mapping) later work has challenged the original methodology even though the replication succeeded. The crowd field driving the fame-trap score is still my estimate rather than logged data.

Same test.No surface clues this time.

Here's how you did.

The base rates you're being compared against

Is this skill even real? The evidence is now split.

Your misses

Same test.
No surface clues this time.