1 · Problem
Candidate production can outrun authoritative evaluation.
Machine cognition can help people produce sophisticated candidate claims outside their own defensible authority. That does not make the candidates worthless, but it does create a routing problem. Domain experts may be the actors able to supply missing warrant, while expert attention remains scarce.
The handoff system therefore needs a pre-filter. Its job is not to declare which ideas are true or important. Its job is to decide which epistemic action should happen next and where scarce expert review is most likely to be useful.
Triage may route authority. It must not manufacture authority.
2 · Closest prior art
The pieces already exist.
Idea screening. Bell, Kavadias and Sommer show that machine learning can screen crowdsourced ideas when expert attention is limited, especially by eliminating candidates experts also consider poor.
Specialist idea evaluation. Gimpel and colleagues explicitly describe expert juries as scarce and expensive and test crowds and LLMs as substitutes or preselectors.
Selective prediction and learning to defer. These literatures formalize abstention, risk/coverage trade-offs and routing uncertain cases to humans.
Active learning. Budgeted query selection already asks where expensive human labels or expert judgments are most worth buying.
Editorial triage. Journals and conferences already protect reviewer capacity with desk screening, and AI-assisted peer-review research increasingly studies automated triage.
Value of information. Decision theory already provides a normative language for asking whether further information is worth its acquisition cost.
The project should claim none of these mechanisms as new.
3 · Candidate state
Route a claim state, not a prestige score.
Each candidate should carry a minimal handoff state:
- the claim or research-object identifier;
- current epistemic status;
- origin and contribution provenance;
- lineage-search state;
- evidence already supplied;
- the explicit authority deficit;
- the epistemic function requested next;
- uncertainty and known failure modes;
- the recipient expertise class, if known.
Originator identity may be retained for provenance while hidden from the triage decision where possible. Institutional prestige should not silently become a proxy for epistemic value.
4 · Routing policy
The output is a next action.
For a candidate c, the routing policy should choose among actions such as:
- escalate to a named expert or expertise class;
- request clarification or a more testable claim;
- run a cheaper check, such as lineage search, computation or formal verification;
- defer until prerequisite evidence exists;
- park while preserving the candidate and auditability.
The system does not have an action named validated. Only evidence, successful formal checks, replication or suitably authoritative review may change the claim into a stronger epistemic state.
5 · Objective
Optimize expert attention against epistemic state change.
First formal target
Maximize useful epistemic state change per unit of expert attention, subject to explicit constraints on false negatives, calibration, auditability, status bias and authority preservation.
This is intentionally not a scalar claim that an algorithm can measure “idea quality.” The first experiments should use a dashboard of constrained quantities rather than compress everything into one score.
6 · Mandatory reject audit
A filter that never inspects its rejects cannot know what it destroys.
Every experimental triage policy should randomly escalate a sample of candidates it would otherwise reject or park. Those audited rejects receive ordinary expert adjudication.
This provides an empirical estimate of the false-negative region and creates a continuous route for surprising candidates to re-enter the system. The audit rate can later become adaptive, but it should not disappear.
“We saved expert time” is not a result unless the missed-candidate cost is also measured.
7 · Test design
Compare the mechanism against simpler baselines.
Use the same blinded candidate corpus across at least four routing arms:
- Direct expert review: no pre-filter.
- Rule-based triage: claim clarity, evidence and lineage-completeness checks only.
- Model triage: model-assisted ranking or escalation.
- Claim-sensitive handoff triage: model/rule assistance plus explicit authority deficit, provenance, uncertainty and mandatory reject auditing.
On a benchmark subset, use multiple independent experts and adjudication rather than treating one reviewer as truth. Randomize or blind originator credentials. Include semantically equivalent rewrites to test robustness against presentation gaming.
8 · Success and retirement
The special mechanism has to earn its complexity.
Continue development only if the claim-sensitive protocol reproducibly improves a constrained outcome such as equal or better high-value recall at materially lower expert time, or more decisive epistemic state transitions per expert-hour without unacceptable false-negative inflation.
It should also avoid a large credential effect and remain reasonably stable under superficial rewriting.
If ordinary editorial triage, a simple ranking model or a cheap rule set performs equivalently, adopt the simpler method and retire the special terminology.
9 · Retrospective-first study
Reuse expert attention that has already been spent.
Agalmic Research does not have the collaborator base to begin with a large prospective expert experiment, and doing so would contradict the scarcity thesis. The first empirical study therefore uses historical peer-review records as inherited experimental infrastructure.
Begin with PeerRead as a compact feasibility corpus, then replicate on suitable public OpenReview venues if the first study is informative. Later, carefully matched OpenAlex metadata may provide downstream citation/topic outcomes without treating citations as truth.
Impose simulated expert-review budgets across the historical candidate pool and compare allocation policies. Report an attention-efficiency frontier rather than one arbitrary threshold. Historical accept/reject decisions, reviewer ratings, confidence and disagreement should be treated as separate imperfect outcomes rather than collapsed into “quality.”
Only if retrospective evidence shows a credible delta should the programme consume new expert attention on prospective validation.
10 · Product boundary
Build the test before the marketplace.
The first product is not a global expert-routing platform. It is a reproducible protocol, benchmark and small experimental harness that can answer whether structured triage is worth using at all.
A later tool might accept candidate objects, run cheap checks, produce a routing recommendation, preserve an audit trail and package the handoff for an expert. A market, reputation system or global expertise directory should wait until the basic allocation mechanism survives empirical testing.