Research backlog and decision log
4 of the 6 closed
hypotheses here failed, and one of them had already shipped before it was tested.
A backlog showing only successes is not a record of research, it is a record of
selective reporting.
Index built 2026-08-10T07:23:47. 1822 passages from 90 arXiv papers and 12 synthetic internal notes. Embeddings: Snowflake/snowflake-arctic-embed-s (ONNX int8, Apache-2.0), loaded from Demo 1. One embedding stack across both demonstrations. Demo 1 harness live model configured
Retrieval, extraction, every statistical test and the backlog are model-free. Only the synthesis paragraph on screen 2 uses a model, and it degrades to extraction.
Retrieval, extraction, every statistical test and the backlog are model-free. Only the synthesis paragraph on screen 2 uses a model, and it degrades to extraction.
Supported
1
Null
2
Contradicted
2
Inapplicable
1
Closed hypotheses
| ID | Hypothesis | Outcome | Effect | 95% CI | p | n | Effort |
|---|---|---|---|---|---|---|---|
| HYP-2026-014 live | Cross-encoder reranking over a fixed candidate pool improves context recall@5 Executed 2026-08-10T07:34:30 through Demo 1's harness (imported), technical-manual question set. This row is written by the run on screen 4, not stored. | not supported — null result | +0.147 | [-0.005, +0.351] | 0.220 | 10 | 1 hour |
| HYP-2026-009 | Semantic chunking will raise context recall@5 by at least 3 points Index build time rose from 11 to 48 minutes for a difference of one point that the interval cannot separate from zero. Rejected on cost. The published negative result agrees, but note it tested a different intervention: semantic boundary detection, not late chunking. Do not cite this as evidence against late chunking. Evidence: ENG-2503-011 (synthetic internal note), arXiv:2410.13070 | null | +0.010 | [-0.040, +0.060] | 0.620 | 140 | 2 days (re-index and re-score) |
| HYP-2026-011 | Carrying the section heading and equipment variant into each chunk will raise recall The cheapest change in the backlog and the largest effect. It was proposed third because it is unglamorous. Worth saying out loud: the ranking of ideas by expected excitement was inversely related to their measured value. Evidence: ENG-2509-014 (synthetic internal note) | supported | +0.080 | [+0.030, +0.130] | 0.004 | 140 | 4 hours (metadata change, no re-embedding of source text) |
| HYP-2026-006 | A larger embedding model will raise recall enough to justify the latency 335M and 570M parameter models were within each other's intervals while costing 3x and 5x the latency of the 110M model. The curve flattens; the latency does not. Standardised on the 110M class. Evidence: ENG-2510-006 (synthetic internal note), arXiv:2210.07316 | null | +0.010 | [-0.050, +0.070] | 0.710 | 140 | 1 day |
| HYP-2026-002 | Retrieving 40 passages instead of 5 will raise answer acceptability Median latency went from 2.1 s to 9.4 s, so it failed the 3 s constraint outright and the accuracy question became moot. Recorded as inapplicable rather than negative: on a GPU serving path the answer could differ, and filing it as a negative result would mislead whoever reads this next. Evidence: ENG-2602-013 (synthetic internal note), arXiv:2307.03172 | inapplicable | — | — | — | 140 | 1 day |
| HYP-2025-021 | An LLM judge can replace human review for the release gate Judge-human kappa 0.47 against human-human 0.61, and reversing presentation order flipped the judge's preference on 19 of 120 pairs. The judge stayed as a screening filter and the release gate stayed on deterministic checks. This is the item that was already half-shipped when it was tested. Evidence: ENG-2507-009 (synthetic internal note), arXiv:2305.17926, arXiv:2306.05685 | contradicted | -0.140 | [-0.240, -0.040] | 0.008 | 120 | 3 days including human annotation |
| HYP-2025-018 | Our reported context recall of 0.81 is real Gold passage IDs had been resolved against the improved index and reused to score the baseline, which could not retrieve IDs that do not exist in it. The true figure was 0.68. This is why `leakage_checks` is a mandatory field on the card rather than a suggestion, and why the number was wrong in the direction we hoped. Evidence: ENG-2511-021 (synthetic internal note), arXiv:2207.07048 | contradicted | -0.130 | — | — | 140 | 2 days of investigation |
Every stored row's numbers come from the synthetic internal notes, which are the only invented data in this demonstration and are labelled as such throughout. The row marked live is written by the execution on screen 4 and is absent until that run has happened.
Correcting for running many hypotheses
| ID | Hypothesis | p | Significant at 0.05 | Survives BH at FDR 0.05 |
|---|---|---|---|---|
| HYP-2026-009 | Semantic chunking will raise context recall@5 by at least 3 points | 0.6200 | no | no |
| HYP-2026-011 | Carrying the section heading and equipment variant into each chunk wil | 0.0040 | yes | yes |
| HYP-2026-006 | A larger embedding model will raise recall enough to justify the laten | 0.7100 | no | no |
| HYP-2025-021 | An LLM judge can replace human review for the release gate | 0.0080 | yes | yes |
Benjamini-Hochberg is used rather than Bonferroni because these hypotheses are correlated configurations of the same system, and Bonferroni is over-conservative there. The correction is applied across the whole backlog, not per experiment.
Open, cheapest first
| ID | Hypothesis | Effort | Test |
|---|---|---|---|
| HYP-2026-015 | Fixing the candidate pool at 10 rather than 20 costs no recall and buys 150 ms Directly downstream of whatever HYP-2026-014 returns. If reranking does not help at 20 it certainly will not help at 10, and this item is dropped. Evidence: arXiv:2411.11767, ENG-2411-017 (synthetic internal note) | 1 hour | paired bootstrap; McNemar on hit@5 |
| HYP-2026-016 | Late chunking raises recall on passages whose meaning depends on earlier context Note the subset must be fixed in advance. Choosing the subset after seeing the results is how a null becomes a headline. Evidence: arXiv:2409.04701, arXiv:2312.06648 | 1 day | paired bootstrap on the subset, with the subset defined before the run |
| HYP-2026-017 | Answer quality is unchanged when the judge is calibrated against 50 human labels The 2026 literature turns LLM-as-judge from a method into a measurement problem. This item is what it would cost us to adopt that properly. Evidence: arXiv:2601.20913, arXiv:2605.16354, ENG-2507-009 (synthetic) | 3 days including annotation | calibrated judge with variance correction, per arXiv:2601.20913 |
Decision log
Nygard's five sections, with MADR's Confirmation field. A decision without a confirmation clause is an opinion with a date on it.
ADR-014 — Reranking is enabled only at a fixed, small candidate pool
accepted
2026-03-02
Context. The literature disagrees with itself. Cross-encoder reranking is standard practice and reported to cut retrieval failure substantially, while arXiv:2411.11767 finds returns diminish and then go negative as the candidate pool grows, and arXiv:2503.08046 finds the advantage inverts as queries get more conditions. Our own note ENG-2502-004 (synthetic) reports a null on short factoid queries. Reranking latency is also the largest single item in the 3 s budget.
Decision. Reranking is permitted with the candidate pool fixed at 20 and the operating point measured. It is not permitted as a way to widen recall by reranking more candidates. Any claim about reranking must state the candidate count.
Consequences — good
- The intervention stays inside the latency budget with room for generation.
- The disagreement in the literature is resolved by naming the condition rather than picking a side.
Consequences — bad
- Recall lost before the reranker cannot be recovered by it.
- A future first-stage improvement changes the arithmetic and this must be re-tested.
Confirmation. The hypothesis card HYP-2026-014 executes exactly this comparison through the shared evaluation harness. Re-run it after any index rebuild.
ADR-011 — The release gate stays on deterministic checks, not a judge model
accepted
2025-08-04
Context. We wanted to replace manual review of assistant answers with an LLM judge. The audit in ENG-2507-009 (synthetic) measured judge-human kappa at 0.47 against human-human 0.61, with a systematic preference for longer answers and a position-order sensitivity that flipped 19 of 120 pairs. arXiv:2305.17926 reports the same failure mode.
Decision. The release gate is: the expected value is present, no forbidden value from another equipment variant appears, and every citation resolves to a supplied passage. The judge is retained as a screening filter over the queue only.
Consequences — good
- The gate is auditable by a maintenance engineer, satisfying the explainability constraint.
- The gate cannot silently drift when a model version changes.
Consequences — bad
- Fluency and helpfulness are not gated at all, only correctness proxies.
- Manual review remains the bottleneck for anything the deterministic checks miss.
Confirmation. Re-run the agreement audit whenever the judge model changes. A kappa below the human-human figure keeps the gate deterministic.
ADR-009 — GraphRAG is out of scope until indexing capacity changes
accepted
2025-06-10
Context. GraphRAG (arXiv:2404.16130) answers global questions our current system cannot. Its index construction runs a language model over every chunk and summarises every community. Costed against our corpus in ENG-2506-002 (synthetic) as a multi-day CPU job, re-run weekly as revisions land, with query-time map-reduce measured at 6 to 11 s against a 3 s budget.
Decision. Rejected under the standing constraint set. Recorded with the costing so it is not re-evaluated from scratch each quarter.
Consequences — good
- Corpus-level summarisation questions are explicitly out of scope, and users are told so.
- The rejection has a stated trigger for revisiting rather than being permanent.
Consequences — bad
- A real capability gap remains open.
- The costing is specific to our corpus size and goes stale as the corpus grows.
Confirmation. Revisit if GPU capacity becomes available for indexing, or if the corpus shrinks enough that the index job fits in a night.
ADR-007 — Gold passage labels are resolved per retrieval arm
accepted
2025-12-02
Context. Six weeks of reported context recall were inflated because gold passage IDs were resolved against one index's chunk boundaries and reused to score another (ENG-2511-021, synthetic). The error flattered the change being justified.
Decision. Gold passages are resolved per arm from the source document span, stored with the question, and any cross-arm comparison asserts that both arms had a resolvable gold set before reporting a difference.
Consequences — good
- Cross-arm comparisons became meaningful.
- The reported baseline dropped from 0.81 to 0.68, which is the true figure.
Consequences — bad
- Label resolution now runs per arm, so evaluation is slower.
- Historical numbers before this date are not comparable and are marked as such.
Confirmation. The evaluation refuses to report a cross-arm delta when either arm has an unresolved gold set.