Results
A useful result can be positive, negative or corrective. This page records what testing, prior work and implementation have forced us to believe more narrowly than before.
A simple rule
A result should change what we are willing to claim or what we do next.
Some items here are working research results rather than peer-reviewed findings. Their status is shown so readers can distinguish evidence from interpretation and open questions.
The emphasis is on what changed, not on the machinery that produced it.
The study reconstructed 1,815 usable ICLR 2020 papers, but that sample was too small to distinguish the planned effects with enough precision. The citation outcomes stayed sealed rather than weakening the test after seeing the answer.
What remains: A clean negative result and a better estimate of how much data a useful successor study needs.
The available unbiased historical sources still fall short of the planned sample. A convenient 2021 source was rejected because its availability depended on the historical decision, which would introduce selection bias.
What remains: The next problem is finding a neutral historical cohort, not writing more analysis code.
In an early expert-attention benchmark using a 20% review budget, ranking by post-review score preserved many historically accepted papers but preserved very few high-disagreement cases.
What remains: A useful allocation policy must preserve more than agreement with a past institutional decision.
Existing work on knowledge transfer, brokerage, absorptive capacity and expert evaluation already covers much of the general problem of moving knowledge toward people able to assess or use it.
What remains: A narrower question: can a concrete handoff protocol make missing expertise and changes in confidence easier to track?
Established economics already contains strong accounts of bottlenecks, complements, production constraints, innovation and rebound effects.
What remains: An empirical question about how bottlenecks, value and rents move when a previously expensive capability becomes cheap.
Software experiments can test specific questions about representation, analyst performance and discovery workflows. They cannot by themselves establish broader economic or institutional claims.
What remains: Use Nemosyne where the intervention and outcome can actually be observed and measured.
OpenReview confirmed that the requested historical PDF is not public, and a later bounded retry of the healthy Common Crawl 2020-45 index found no captures for any of ten frozen forum URLs or their ten corresponding PDF URLs.
What remains: The study now moves to a prospectively frozen source-comparability design, beginning with a provenance audit of PeerRead’s ICLR 2017 paper drafts. Citation outcomes remain sealed.
PeerRead looked like a possible ICLR 2017 source, but its preserved papers are not clean submission-time snapshots: one accepted PDF directly contains material its meta-review says was added after reviewer discussion. Once that source is excluded, the largest remaining admissible addition leaves the study at 4,547 candidates, below the frozen 4,600 floor before further losses.
What remains: A terminal outcome-blind result. The citation outcomes remain sealed, and any continuation must be a separately frozen v0.3 source design rather than a relaxed interpretation of v0.2.
In a frozen synthetic hold-out over 7,531 ICLR papers with at least three scored reviews, disagreement beat random allocation consistently but by less than the preregistered materiality threshold. At a 20% budget it captured 22.4% of total hidden-review update mass versus 19.5% for random.
What remains: The objective matters. A simple near-boundary rule captured 65.3% of cases where the hidden review moved the aggregate across the recommendation midpoint, suggesting that decision-instability and general opinion change require different allocation policies.
Across 20 new deterministic hold-out splits, prioritising papers whose two observed ratings averaged closest to the recommendation midpoint reached 95% pooled midpoint-crossing recall while saving a median 45% of simulated third-review work. Random routing saved essentially none at the same preservation target.
What remains: The simple rule, not a learned allocator. The result is retrospective and pooled across years, so any operational follow-up must use sequential review state and enforce preservation floors within each rating regime.
Why keep negative results?
If a study is too weak to answer its question, if prior work already explains a proposed idea, or if a convenient dataset would bias the result, recording that limit prevents wasted effort and overclaiming.
The aim is not to make every idea survive. It is to make the surviving ideas better.