An Alphabetical File Listing Outlasted Most of My Ideas

The short version: on a rotated benchmark with new tasks and a repaired harness, a plain alphabetical file listing beat the context maps I built for small local models. After that, a hindsight oracle (one that picks the best approach for each task after seeing the scores) couldn’t clear the bar I had set for building a task router, a classifier to pick the approach in advance. The experiment stayed inconclusive, but under the rule I had written beforehand the engineering decision was clear: stop rather than build the router. Along the way my own intuition about which task needed which context showed no evidence of beating random picks. The only piece of the original architecture that survived was a small provenance checker. If you are weighing a context platform or router yourself, the three questions I would now ask first are near the end.

It started as a weekend tangent and took ten days and 333 local-model runs. The intuition was that small models spend much of their limited capability rediscovering the environment (which repo has the config, where the decision was recorded), so my software factory (the agents and scripts that do most of my build work) could do that ahead of time and hand the model a prepared briefing. The day-one design had a provenance-stamped SQLite projection with full-text search, a context compiler on top and a graph database in reserve. It had a diagram. The diagram had arrows. It also had one rule, written before any of it got built: the cheap alternative gets measured before the expensive one is built.

The cheap test first

Everything ran locally on one 24 GB MacBook Air through Ollama, qwen2.5-coder:7b to start and devstral-small-2:24b (Q4_K_M) later, over a universe of ten personal repositories. No frontier model answered any task. The bar, written down in advance, was +0.15 on a 0 to 1 task score; anything within 0.05 was treated as grading noise. The grading was not independent. The same programme wrote the tasks, the rubrics and the context arms. The first experiment was graded unblinded and every later one with the arm labels hidden, by the same programme that ran them. By my own rule about judges, that is an uncalibrated grader, and the 0.05 noise band was a setting I chose, not something I measured. For the experiment the decision rests on (E3a), I later ran the independent check; the results are below. The earlier experiments were not re-graded.

The first experiment asked whether a machine-built repo map helped a 7B model at all. It managed +0.084 over plain access. A control arm got hand-curated context instead (structural pointers written by an expert agent, no conclusions), and that lifted the same model by +0.320, or +0.233 once I dropped a task where a packet had leaked part of the answer; single runs per cell, so a first look. A second, better map with three replicates per cell scored +0.094. The graph-ranking term, the PageRank bit I was privately proudest of, added +0.012 when kept, inside the noise, so it came out. The record says dead as built: that map, on a 7B model.

The 24B detour, and a directory listing

This is where the story tried to go well for me. E1c ran the same map against the 24B model and cleared the bar: +0.20 over a plain file listing, on its own benchmark. I had a narrative and everything. I also had a harness bug that stored the model’s last tool call as its final answer at the turn limit, which ate 20 of E1c’s 54 runs across all arms. E1c’s recorded numbers were left as they were, not corrected for it; the repaired harness went on to produce clean answers in 78 runs.

E2 asked again with new tasks, the repaired harness and an improved map. On the main tier (the three tasks the benchmark was built around, two of them repo-wide sweeps) the plain alphabetical listing scored 0.726; the machine maps scored 0.390 and 0.431. The first map lost to a directory listing by 0.336 and the improved one by 0.295, and the listing also led both maps on the average across all six tasks. It didn’t win everywhere, and it had costs: on the next tier of tasks the improved map beat it by 0.15, right at the bar, and across the grid the listing made more unsupported claims than the improved map (0.72 per task against 0.39) and missed more of the expected evidence.

The raw answers suggest why, and this is the part I’d show anyone who puts a repo map in front of a coding agent. Two of the main-tier tasks were sweeps: find every repo that does X. On both, the ranked packet pointed the model at a few highly ranked files; the model read them carefully and concluded that only a handful of repos did X (on one task, the same wrong answer three seeds running). The listing’s only opinion was the alphabet. It named every repository, unranked, and the model swept them. In these runs, ranking narrowed where the model looked, and the narrowing went with consistently wrong answers. That is one 24B model on two sweep tasks.

Hand-curated context still helped for a fourth cycle, but on that same tier the plain listing matched it, 0.726 to 0.702, inside the noise. The directory listing had started to feel like the most reliable engineer on the project.

Three limits. The listing was never complete. To fit a fixed budget each repo got an equal share, roughly 20 files, so it named every repository but left most files out, and every arm, maps included, had a tool to list directories; what differed was the packet, not access. Its alphabetical cut-off happened to surface each repo’s .hekton/ files first (including the project config for 7 of the 10 repos), which is where at least one sweep task’s answer lived. At monorepo scale even a capped listing of every repo stops fitting, which is exactly where ranking ought to earn its keep, and I didn’t test that. And E1c’s +0.20 stands as measured: E2 changed tasks and harness together, so task shape is the likeliest explanation for the difference, not a proven one.

Asking the oracle first

The curated packets looked less like better facts and more like better instructions about how to look: list everything for some tasks, go straight to the likely file for others. So the next idea was to route each task to the right exploration procedure (plain listing, ranked packet or procedural template) with a classifier. Before building the classifier, I measured the best it could ever do.

E3a ran twelve fresh tasks (nine main ones and three controls) against all three arms, then let an oracle look at the scores afterwards and pick the best arm for each task. Any classifier choosing among those arms can only match that or do worse. The rule, written first: if the oracle can’t clear +0.15 over the fixed defaults, nothing gets built. I also wrote down an inconclusive band, [0.10, 0.20], for the comparison against the template, so a borderline number couldn’t be rounded my way. 99 runs, zero infrastructure failures, graded with the arm labels hidden, and at first no grader cross-checked against a second one.

Pooled over the nine main tasks, the oracle gained +0.10 against always using the plain listing, below the bar. Against always using the template it gained +0.16: above the bar, inside the band I’d set for that comparison. The verdict is inconclusive, and I’m not rounding it for a headline. Picking the best of three noisy scores after the fact also flatters routing. A held-out version, which chooses each task’s arm from the other replicates and scores it on the one left out, is a fairer stand-in for a real router: it earned roughly +0.00 to +0.05 over always using the listing. And when I shuffled the arm labels at random within each task and replicate, the hindsight oracle still found a lead over the listing of about +0.10: roughly what the real one showed.

A best-of-three pick looks good even when there is nothing to pick.

Then I checked the grading. Every E3a answer went to fresh sessions of the original grader’s model family that saw only the task, the rubric and the answer, and separately to a second model family under the same conditions. Both used my original rubrics, neither could check an answer against the repositories, and no human graded. Individual scores moved (the graders disagreed with the original by 0.05 to 0.06 on average; neither changed which arm led, though both were slightly kinder to the template relative to the other arms), but the decision did not: the oracle’s lead over always-plain came out at +0.11 under both, below the bar, and the rule returned inconclusive every time. The original oracle had cleared +0.15 over the template by a narrow margin. Neither independent re-grade did: the lead fell to +0.133 and +0.131. The rule required clearing the bar against every registered default, so the decision stayed the same. The hidden labels had not hidden everything, though. A reviewer shown only the answers could name the arm on every convention-audit answer, because the template’s own vocabulary leaked into them; on the sweep tasks it did no better than chance.

A separate audit of the design found biases, most of them pushing toward a pass: the template arm got a short generic procedure and no file map while the other two got a map of files, ranked packets sometimes contained a rubric answer outright, and nine audit-style tasks, three per predicted shape, is about the friendliest mix a router could ask for. The oracle still didn’t clear the bar. One bias ran the other way: the same capped listing as in E2, whose cut-off favoured the plain arm on some tasks. So E3a was not a clean comparison between complete visibility and selective ranking.

Measure the ceiling before funding the approximation.

The finding that stung was smaller. Before the run I had written down which procedure each task shape needed. That mapping matched the oracle’s actual winners on 2 of the 9 main tasks (ties go to the listing, then the ranked packet), or 3 if a tie counts as a match, and 2 or 3 under the independent graders by the same rule (up to 4 if ties count). With three arms, random picks would land about 3. So there is no evidence that my intuition about which task needed which context, the intuition the whole router would have encoded, beat chance. Nine tasks can’t say more than that.

So no classifier, and I chose to stop rather than run a bigger grid to settle it. Three replicates per cell still left wide variation, in some cases from 0 to 1, and a 90% bootstrap interval over replicates for the oracle’s lead over the listing runs from about +0.05 to +0.20 across the three gradings, either side of the bar. On these tasks and candidate strategies, the evidence did not justify building the router under the rule I had written down beforehand. This supports the engineering decision to stop; it does not statistically prove that routing cannot help.

The router question stayed inconclusive; the broader claim did not. After the independent check I recorded the underlying claim (that mechanised context compilation reduces frontier-model dependence) as disproven, narrowly. The hypothesis had its own disproof criterion, written at the start: a materially simpler mechanism achieves equivalent results. E2 met it most directly: on rotated tasks, the plain listing beat the machine maps I had built, though E2 was never re-graded. E3a pointed the same way, with always-plain scoring at least as well as always-ranked under all three gradings, and the router that might have rescued the idea did not clear its gate. That met the narrow disproof criterion for the mechanism I had proposed, not for context engineering in general, and “curated context helps” is kept.

The pooled number hides a clean win and a cost, and both survived the re-grade. On one needle-hunting task the ranked packet scored 1.000 in all three replicates against 0.333 for the listing, though its packet named the file and the exception class outright. On one census task two of three ranked runs stated that several repositories lacked a file they actually had. (The original grading had ranked packets carrying the most unsupported claims across the grid. Both independent graders put the plain listing highest instead, counting only claims the rubric could judge, so I’ve dropped my claim: under both, the plain listing had the highest pooled score and also the most false assertions.) And the procedural template, my attempt to bottle the curated packets’ advice about how to look, pooled below the plain listing under every grading, but it was also the under-dosed arm, so that says less about procedure than it looks. One clean win and a measurable cost make a watch item, not a product.

What survived, and what to ask on Monday

The lab had a second question, run separately and not currently sharing code with the first: can records scattered across repos be joined with full provenance, and say honestly when they’ve gone stale, with no model involved? The first real test answered one cross-source question with all 45 matching records, against an answer I had worked out by hand before writing code, and said “unresolved” where a source had no identity for the repo instead of inventing a link. Its verdict was only partial, for the reason this whole post keeps arriving at: for that one question, about ten minutes of manual grep was not meaningfully more expensive than building the thing. Its advantage was never one-shot speed. It was what happens after the sources change, when every receipt can be re-checked mechanically. I did not compare that against the cheapest obvious alternative: save the one-off grep as a script and rerun it later. A drift test edited one cited file and deleted another: the 2 affected records went stale, 43 stayed fresh, and nothing stale was reported fresh. A flat file matched SQLite’s answers (once run-specific fields were normalised) with 81 lines of storage code instead of about 230. At extraction time, on 27 September 2026, organic use stood at 4 of the 8 genuine uses its 30-day gate asks for, all four logged on 20 September. Useful, narrow, unproven. Even the part that survived has earned another baseline before it earns a platform.

What survived still has to beat a saved grep script before it earns a budget.

Baseline comparisons and oracle analysis are old ideas; what I’m describing is using them as a funding stop rule with the bar written down first, which I call an oracle gate for want of a better name. If someone brings you a proposal for a context platform or a task router for coding agents, ask three questions before you fund it.

1. How does the cheapest unranked mechanism that still names everything in scope do on the team’s real tasks? For my ten personal repos that was an alphabetical listing naming every repository, capped to fit a fixed budget, plus a tool to list directories. For a codebase too big to list in context, I don’t know what the right cheap baseline is (a bounded directory walk, a filename inventory, unranked search results) and nothing here tested it.
2. How does hand-curated context do? That is the target a machine is trying to reach.
3. What does a hindsight oracle over rough versions of their candidate strategies say? It gives two numbers that answer different questions. The raw ceiling, what the oracle gains over the simplest default, asks whether there is enough theoretical upside to keep investigating; if it can’t clear a bar written down before the results, stop there, and check it against shuffled strategy labels, because a best-of-several pick looks good even when there is nothing to pick. The held-out gain, choosing each task’s strategy on some runs and scoring it on others, asks whether a real selector could actually earn that upside, and it is the number to fund on. Here, under the original grades, the raw ceiling cleared the bar against one default while the held-out gain over the listing was at most about +0.05.

None of the three is free: the second needs someone to curate context for real tasks, and the third needs each candidate strategy to exist in rough form. I’d expect them to cost less than building the platform to find out, though I haven’t measured that. It’s the same instinct as routing on evidence rather than on how expensive something feels, one level up, and it’s why I now treat designed context as the thing to beat rather than the thing to automate away.

None of this says graphs, retrieval, bigger models or task routing don’t work. It says what happened on these tasks, with these models, at this scale, against bars I wrote down before I knew the answers.

I rebuilt that surviving pattern from scratch, on invented data, as a small repo: where-from. It joins a service catalogue with architecture decision records and keeps a receipt for every fact. Clone it and run the drift demo:

git clone https://github.com/dermdunc/where-from
cd where-from
./examples/drift-demo.sh

The demo builds a snapshot, then edits a cited file, deletes another and hands a service to a new team underneath it. Its second check ends like this, one line per fact: status, fact id, field, and the record it came from. The missing lines mean a file the ADR cites was deleted (the ADR itself is intact), and the unverifiable line is a cited URL that was there before any drift:

  stale        f-70bd44e08e  search.owner  catalog.json#search
  stale        f-089326a4c0  search.tier  catalog.json#search
  stale        f-f6b1be2f9c  search.lifecycle  catalog.json#search
  stale        f-5243b6db77  search.repo  catalog.json#search
  stale        f-da4c0b550b  checkout.decision  adr/0001-retry-card-payments.md
  unverifiable f-6747002b0d  search.decision  adr/0002-search-index-rebuild.md
  missing      f-fe77c2466b  checkout.decision  adr/0003-email-via-notify.md
  missing      f-1ba41b924c  notify.decision  adr/0003-email-via-notify.md
fresh=8  missing=2  stale=5  unverifiable=1

The demo is the smaller half. The larger half is in the repo’s methodology and what didn’t work, including what would justify reopening each closed idea. The experiment itself can’t be re-run from it, because the tasks came from my private repos, but the E3a per-run scores under all three gradings and a small calculator for the oracle gate are in the repo.

The alphabetical file listing, which I had included only as a baseline, lost one experiment’s headline comparison to a map by more than the bar, in E1c, and came back ahead on the next. I promised myself an independent re-grade before publishing this, and it didn’t change the decision. It did take one of my favourite sentences out of the post.