CodeNib on ContextBench: 77% Line Recall Before Repair

CodeNib's search-only stage reached 77.07% line recall on 500 ContextBench tasks, using two workers that explored source without editing code or running repairs. They used iterative file discovery, grep and source reads, with recorded search model usage of about $0.0114 per task. The complete search-and-repair system resolved 388 of 500 tasks.

Submitted · Leaderboard review pendingThese are our submitted results; an official ranking has not been assigned.

Search line recall

77.07%Before repair

Final line recall

77.90%Including confirmed repair reads

Native Pass@1

77.60%388 / 500 resolved

Search usage / task

$0.0114Two Haiku workers; model usage only

The search workers use Claude Haiku5.5; Claude Opus5.5 handles repair through an adapted mini SWE-agent loop. The public submission contains the configuration and evidence. All 500 task annotations had been exposed during development, so the result describes that known population.

Most annotated coverage is found before repair

ContextBench annotates source code relevant to repository issues. Line recall measures how much of that annotated code the system retrieves. Our 500-task verified split spans eight languages and four source benchmarks. We average each task's scores, keeping all 500 tasks in the denominator.

Adding confirmed original-source reads from repair increases recall by 0.83 percentage points. The search phase already supplies most of the eventually covered annotated code. This is a comparison of stages in one pipeline; there is no search-only Pass@1 result.

Search-only here means two agents repeatedly choosing where to search and what to read, before any patch generation. Its context is the union of source actually observed by either worker. That cumulative search context can be much larger than the compact navigation message passed to the repair model.

Fixed role routing, adaptive search inside each worker

CodeNib's coordinator decides which role runs next and what information crosses model windows. In this run, that routing is fixed: first Haiku search → complementary Haiku scout → Opus repair. Within each search, the model chooses its next tool call from the evidence it has just read.

1. First Haiku workerOriginal issue → plan → glob / grep / read → observe → next plan
2. CodeNib coordinatorObserved file/range locations → versioned navigation, capped at 2,048 tokens
3. Complementary Haiku scoutOriginal issue + navigation → its own search/read loop to investigate gaps
4. CodeNib coordinatorAuthenticate and union observed source → navigation for repair
5. Opus repair agentFetch needed source → edit → self-test → final patch for native grading

This is role routing with two search workers on every task. The submitted system does not use an issue classifier to choose Haiku or Opus as the search model. The 500-task result therefore cannot establish a gain from dynamic model selection.

1. The first worker follows evidence through the repository

   The issue supplies an initial hypothesis. glob finds likely files; grep locates names and uses; read exposes implementation behavior. New identifiers, called helpers and unanswered questions drive the next search. The prompt asks for relevant functions, callers, configuration, interfaces and validation tests, with up to 20 worker turns. The worker can revise an unproductive query and continue a truncated read.

2. The coordinator creates a map of observed locations

   It merges duplicate or adjacent observed ranges and encodes file paths, line intervals and handles bound to the source version. At most 2,048 tokens of this navigation enter the scout's prompt, alongside the unchanged issue. Omitted intervals are counted explicitly. Source bodies and the first worker's generated explanation stay outside this handoff.

3. The scout searches for missing evidence

   The scout gets a separate model window and an explicit complementary role: investigate connected entry points, callers or callees, alternate paths, tests and configuration that the first search may have missed. It reads source itself whenever it needs to understand behavior. Earlier locations are anchors; revisiting them is allowed, and the navigation is explicitly incomplete.

4. The coordinator unions observations and hands off to repair

   After authenticating both workers against one source version, CodeNib unions their observed intervals. Overlapping lines count once. It creates another bounded navigation message from this combined search context for Opus. The repair agent then requests source through codenib-read and codenib-grep, edits code, and runs self-tests through the container shell.

The navigation cap limits metadata at model boundaries. Source bodies read later have their own tool limits; they do not share a total 2,048-token source budget. For evaluation, a location or summary alone earns no source credit: the source must appear in a confirmed model input. The final selected context retains search observations and adds confirmed original-version repair reads. This is a system-wide record of observed source; Opus does not receive the whole union as an initial code dump.

Inspect the pinned search instructions, scout navigation and source union, and repair handoff.

Why this can cover more relevant code

Search adapts to code that has actually been read. An issue may name a symptom while the relevant helper uses different vocabulary. Following identifiers, interfaces and call sites gives the model new search terms beyond the original issue text. Caller relationships in this route are investigated through source tools; no graph expansion is enabled.

The second window has an explicit coverage task. Its job is to look for evidence missing from the first investigation. The coordinator preserves both workers' observed source, so an additional relevant span can add coverage without displacing an earlier one. Deduplication prevents double counting; it does not prevent workers from reading the same code.

The search record retains cumulative reads. It has no fixed top-k chunk selection or 4K final-source cap. That supports broad coverage, with a visible precision tradeoff: search-only line precision is 20.52% and F1 is 28.16%. These scores measure overlap with annotations; they do not establish that every other line read was useless.

Those are design reasons to expect broader coverage. The full 500-task run evaluates them together. In a separate 32-task search study, one Haiku worker reached 33.98% line recall, two independent searches reached 38.82%, and the complementary pair reached 40.00%. The complementary pair's advantage over a plain repeat was +1.17 points, with a 95% repository-cluster interval of [−2.41, +5.12]. The study supports investigating a second search window; it does not establish a stable extra gain from the complementary policy or metadata alone. Those 32 tasks are a different population from the submitted 500.

The measured route uses file discovery, grep and reads throughout. Embeddings, Jev reranking and graph expansion were outside it. The result measures the complete search/repair configuration, including its models, prompts and budgets.

388 resolved, with all 500 attempts counted

Native Pass@1 is the fraction of attempts whose final patch passed the pinned task-specific grading criteria. It does not mean every test in each repository passed. All 112 unresolved attempts remain included. The evaluated tasks span C, C++, Go, Java, JavaScript, Python, Rust and TypeScript.

The overall resolution rate is 77.60%. The source groups show where the outcome varies: 152 of 174 SWE-Bench-Verified tasks were resolved, compared with 34 of 54 SWE-Bench-Pro tasks. These groups have different repository, language and grading compositions; their differences describe this run rather than isolate a benchmark or language effect.

Source: frozen result breakdown, which also includes all eight language slices and their sample sizes.

The context scores expose a tradeoff that the resolution rate alone would hide. Final line recall is high at 77.90%, while line precision is 20.39%. The agent found much of the annotated code, but its selected context also extended well beyond those annotations. F1 remains 28.11%; useful surrounding code may be unannotated, but broad reading is still a reason to inspect precision alongside recall.

Precision measures the share of selected source overlapping the annotations. F1 balances recall and precision; all context means average per-task scores, including per-task F1. A high search recall and a high repair rate are distinct outcomes. Without a matched repair comparison, this run cannot assign the 77.60% resolution rate to one retrieval component.

Almost all model usage goes into repair

Recorded search-plus-repair model usage totals $338.88, or $0.6778 per task. The two search workers account for $5.68 across the run; repair accounts for $333.20, or 98.32% of recorded model usage. Failures and stopped attempts stay in both the cost and outcome totals.

In this configuration, making search cheaper on its own would affect a small part of the total. A more useful next question is whether better source selection can reduce the repair work while preserving resolution quality. That needs a controlled repair comparison; these results do not answer it yet.

The cost accounting uses recorded SDK model-usage values, including caching. It excludes compute, native-test containers and earlier strategy development, and is not a reconciled cash invoice.

How to read and verify this result

The submission is under review in the official ContextBench repository. Its format checks, aggregation from public inputs and replay of recorded inputs pass. Maintainers still need to review task coverage, source attribution and native-grading comparability before leaderboard publication. We are reporting the submitted measurements here without assigning CodeNib an official rank.

All 500 task annotations had been exposed during development. Three original development repairs are reused, with their selected-context exports assembled retrospectively; the other 497 retain their first repair executions. Reference fixes, native test patches and grading feedback stayed outside model inputs. Arbitrary shell-output source attribution remains partial, so the context scores describe the selected source supported by the submission's delivery evidence, rather than a complete inventory of everything the solver may have read.

The public package includes the exact task list, pinned evaluator and implementation versions, outcome records, usage receipts, and raw search and repair trajectories. Its aggregation and report checks run without model keys or Docker. Replaying those records verifies the submitted accounting; it does not rerun inference or the native tests. See the reproduction guide and review notes for the full protocol and limitations.

Explore CodeNib

CodeNib provides repository context and typed code navigation for coding agents. To see source-linked explanations in the browser, open the Wiki demo. To connect the released CodeGraph tools to Claude Code or Codex, follow the quickstart. The ContextBench agent configuration described here is a separate benchmark setup, available in the submission evidence.