Let a Model Plan grep, Then Let Jev Rank the Code

Separate retrieval from judgment

Code retrieval usually has two stages. Retrieval selects candidates from the repository; reranking reads the question and candidate code, then puts the more useful results first. A reranker can only work with the code it receives. If the target function never enters the candidate pool, better ordering cannot recover it.

Jev's Decisions interface fits the second stage. We ask one relevance score question per candidate, using four criteria ranging from irrelevant to a direct implementation or likely fault location, then sort the scores in Python. Our code manages candidates, batching, and failure handling; the model supplies judgments.

BM25 can therefore feed Jev directly, as can code found by grep. Both routes can operate without embeddings, but both still need a way to select candidates from the repository. Sending every chunk to the model moves that selection work into model calls.

Our dedicated reranker baselines were Qwen3-Reranker-0.6B and 4B. They score query–code pairs; Jev returns relevance judgments through a hosted, typed-question interface. All three can sit behind the same candidate interface. We do not assume that Qwen was trained specifically for code, and we do not ask Jev to produce a ranked list of IDs as a chat completion. See the Jev integration guide for configuration.

Three different limits matter here:

A batch size of 10 is our integration setting, not a claim about the API's maximum number of questions. Returning five results does not reduce the number of candidates that need scoring.

Dataset and comparison setup

We used Hugging Face's CodeNib Base, pinned to revision 4eb84e2e8918474969ce68c5b06facf14d6be604. The complete test split contains 100 issues, 25 repositories, and five language groups: C/C++, Go, Python, Rust, and TypeScript/JavaScript. There are 151 target code blocks.

Each repository was checked out at the issue's base_commit, before the fix. The query was the full issue text; models received no patch, hints, or target locations. Candidates used CodeNib's production tree-sitter chunking rules. Each candidate exposed at most 3,000 characters to the model, and hits counted only within the visible line range.

We froze two candidate pools and scored each with all three rerankers. Within a route, the models received identical candidates and text. The grep plans were also reused across models. This produced 100 × 2 × 3 = 600 reranking observations, with one run per combination.

The primary metric is macro-averaged code-block Recall@5. We remove overlapping result spans, take the first five, measure the fraction of target blocks hit for each issue, then average equally over all 100 issues. A hit requires overlapping line ranges in the same file. This measures localization, not the fraction of issues solved. All quality figures below use labels aligned to base_commit; we explain the correction later.

Qwen ran on an H100 80GB in BF16, with batch size 8 and a 12,288-token input limit. The evaluation implementation computed output logits only at the final position, and all inputs fit within the limit. Jev requests pinned typesafe/jev-1.13; successful responses resolved to typesafe/jev-1.13-20260917. Requests ran sequentially in batches of 10 candidates. The timings describe these deployment configurations, not an intrinsic speed ranking of the models.

How deep does BM25 have to go?

BM25 ranks code by word overlap with the issue text, and on these issues it often ranks the target code far down the list. Before calling any reranker, we measured how many BM25 candidates it takes for the target code to enter the pool.

The first 50 candidates contained 54.25% of the target blocks and the first 100 contained 63.78%. It took 1,000 candidates to reach 90.80%. At K=100, 32 of the 100 issues still had no target block in the pool.

Model-planned grep is the contrast. It returned 17.47 candidates on average (median 13, p95 52) and still covered 75.83% of the target blocks, more than BM25's top 100. The cap of 100 rarely came into play for grep. For BM25, 100 is already a long list to rerank, and it is still not deep enough.

Going deeper is expensive for a pointwise reranker. Batching bounds memory and context use, but every candidate is still scored against the full issue. On one Prometheus issue with 14,854 characters of issue text, Qwen 4B took 79.15 seconds to rerank 100 BM25 candidates.

We therefore capped reranking at 100 candidates for both routes. K=1,000 was a BM25 coverage audit only; no model reranked 1,000 candidates in this experiment. The cap reflects reranking cost, not a finding that 100 is the right depth for BM25.

Let a model plan grep, then rerank the code

An alternative is to ask a model to turn the issue into search clues likely to appear in the repository. Our pipeline was short:

Full issue + repository name + source-directory overview
    → One Sonnet 4.6 planning call, producing up to 6 regex/glob actions
    → Run rg locally; map matches to the smallest enclosing visible chunk
    → Round-robin across actions, deduplicate, and retain at most 100 candidates
    → Rerank with Qwen or Jev and return the top 5 results

The planner did not see retrieved code or revise its plan after reading search results. This was a single planning call. Each action also had per-file and total matched-line limits. Empty candidate pools stayed empty: there was no BM25 refill and no answer-informed repair of the search terms.

As the coverage comparison above showed, this finds more target code than BM25 top-100 with about a sixth of the candidates, which also cuts the work passed to every reranker.

It recovered 19 issues that BM25 missed completely, while missing six that BM25 had found. The routes are complementary. For 13 of those 19 recovered issues, BM25's top-100 already contained all the target files. Much of the improvement came from finding the right function inside a file that retrieval had already identified.

Failures remained in the evaluation: one issue had no candidates, and one generated action contained an invalid NUL-matching regex. We scored the actual outputs rather than repairing them using the answers.

Changing retrieval had a clearer effect than changing rerankers

With the aligned labels, code-block Recall@5 was:

All three models benefited from the grep candidates. For Jev, the paired 95% interval for the route improvement was [+5.16, +19.40] percentage points. We used 5,000 bootstrap samples clustered by repository to preserve within-repository issue dependence. Both Qwen route-improvement intervals were also above zero. This is exploratory analysis without a multiple-comparison correction.

Within the same grep pool, Jev exceeded 4B by 3.33 percentage points, with an interval of [−0.18, +7.59]. In the BM25 pool, the difference was 1.05 percentage points, and its interval also crossed zero. Jev had the highest point estimate, but this sample does not establish a consistent advantage over 4B. It does not establish equivalence either.

The models also made different mistakes. On gin-gonic__gin-3820, Jev ranked the target setWithProperType() third, while 4B's top five favored multipart helpers and tests. On sympy__sympy-13031, Jev moved the sparse-matrix row_join() into fifth place. On nushell__nushell-13831, however, Jev overemphasized related row-splitting code, while 4B ranked the target column-splitting function better.

These differences support keeping the reranker replaceable. Latency, API cost, and deployment constraints also matter when choosing one.

Does grep still need a reranker?

If model-planned grep already finds the right code, one might return its first five matches and skip reranking. We scored that option too: the same grep candidates, in the order grep produced them (round-robin across the planned actions), with no model reading the code.

Jev's gain over grep's own order is 12.78 percentage points, a 21.8% relative improvement, with a 95% interval of [+6.44, +19.67]. Part of the pool cannot move at all: 27 issues had five or fewer candidates, so any order returns every one of them, and the gain comes from the issues where grep matched more code than fits in five results.

Reranking is also the cheap part of this route. Jev scored the grep candidates in 0.480 seconds at p50, for $0.045 across all 100 issues. The planning call that produced the candidates took 4.044 seconds at p50 and cost $1.004. Grep narrows the search; the reranker decides which of the matches come first.

Like the rest of this post, these figures use labels aligned to base_commit. The alignment moved grep's own order by only 0.20 points but Jev by 2.17, so the reranker's measured gain is smaller in this frame than it was with the published coordinates.

Why correcting line numbers changed our conclusion

An audit of the label producer revealed a coordinate mismatch: modified symbols were stored with post-fix line numbers, while deleted symbols used pre-fix line numbers. Retrieval operates on the pre-fix base_commit. Comparing retrieved spans directly with the published labels mixes coordinates from two revisions.

For example, inserting code above a target function moves its line range down. In the old revision, those same line numbers may point into a different function. A line-overlap metric can then reward the wrong code or miss the correct implementation.

We relocated all 151 target symbols in base_commit using the label producer's extraction rules. 119 line ranges changed across 77 issues. One duplicated symbol name was ambiguous; we retained the producer's existing dictionary-overwrite behavior and recorded both candidate locations in the audit.

No query, candidate, plan, score, or ranking changed, and we made no new model calls. We recomputed only the hit metrics.

That was enough to change the interpretation. With the published coordinates, Jev's advantage over 4B on grep candidates was +3.83 percentage points, with a 95% interval of [+0.49, +7.83]. After alignment, it became +3.33, with an interval of [−0.18, +7.59]. We withdrew the initial conclusion that this run had established Jev's superiority over 4B. The improvement from changing to grep candidates remained supported after the correction.

The audit fixed revision coordinates. It did not establish complete semantic labeling of every target symbol or rule out model training on these public historical issues. Both the original and corrected metrics remain in the results files.

Fewer candidates, but what about latency and cost?

The reranking stage became much faster: Jev's median fell from 3.010 to 0.480 seconds, and 4B's from 5.101 to 0.717 seconds. But grep added a planning call. Successful planning calls had a p50 of 4.044 seconds, and local grep plus code extraction took about 33 milliseconds at p50.

The table includes both reranking times and sums of stage times. For each route, we sum separately measured component times per issue, then take the p50. We do not add the component medians, and these are not observed production end-to-end request times.

The sums exclude Git snapshot preparation, chunking, index construction, model loading, and file output. They also exclude the initial failed planning calls and offline recovery delays. At four concurrent planning requests, 38 succeeded and 62 returned HTTP 429. We retained the successful plans and reran the remaining requests serially to obtain all 100 plans. The table therefore cannot predict user waiting time under production rate limits.

In this configuration, reducing 4B's candidate count saved enough computation to offset planning. For Jev and 0.6B, the BM25 route remained faster. Removing embeddings does not automatically remove latency; an additional planning call belongs in the budget too.

Recorded API costs for the 100 issues were:

Grep reduced Jev's scoring cost. Including planning, however, the recorded cost was about $1.049 per 100 issues for grep → Jev, versus $0.289 for BM25 → Jev. We did not convert the local Qwen GPU usage into dollars.

Hosted-service availability also mattered. The two Jev routes had 14 and one HTTP 403 responses, respectively, with upstream Cloudflare markers; the exact trigger is unknown. For failed batches, we preserved the candidates' original order and placed them after successfully scored candidates. Those fallback results count toward the quality metrics. Costs for failed requests without usage records are unknown, not assumed to be zero.

Connecting it in CodeNib

In a checkout containing the Jev integration, with retrieval dependencies installed and OPENROUTER_API_KEY configured, BM25 top-100 can feed Jev like this:

from codenib.model.retrieve_rerank_pipeline import (
    RetrieveRerankPipeline,
    RetrieveStageConfig,
)

pipeline = RetrieveRerankPipeline(
    repo_path="/path/to/repository",
    index_path="/path/to/index",
    retrieval_mode="sparse",
    retrieval_plan=[RetrieveStageConfig(engine="sparse", top_k=100)],
    rerank_strategy="decisions",
    rerank_model="typesafe/jev-1.13",
    rerank_candidate_top_k=100,
)
results = pipeline.query("Where are failed payments retried?", top_k=5)

Both the retrieval-stage top_k and the reranking candidate cap are set explicitly. Increasing only the latter will not make retrieval find more code. We pinned the model version for the experiment; the integration also accepts ~typesafe/jev-latest. The integration guide covers installation and model configuration. Model-planned grep remains an experiment script, rather than a default retrieval route.

A combined candidate pool is a concrete next experiment. In a post-hoc audit, we interleaved BM25 and grep results, deduplicated them, and kept the cap at 100. The union reached 83.65% target-block coverage. We have not reranked this pool, so 83.65% cannot be reported as Recall@5. It does suggest a way to preserve the targets each route finds within a fixed scoring budget.

As for the original question about skipping embeddings, this run demonstrated two workable routes without them and measured their tradeoffs. The full 100-issue experiment has no dense or hybrid control, so it cannot establish a general replacement for embedding retrieval. That comparison needs the same candidate budgets, timing boundaries, and aligned label coordinates.

Data and reproduction

The experiment used main commit 61a9ab2fd2cc8f656fa541d6891289f30531f2e7 plus the Jev integration. The implementation, evaluation scripts, and research records are available at the merged CodeNib commit 06f90541.

In that source checkout, docs/experiments/jev_candidates.md records the full method, model settings, failures, and reproduction commands. Its companion jev_candidates_results.json retains both label coordinate frames; this article uses base_aligned_sensitivity for quality. The per-issue results are in jev_candidates_cases.csv, where aligned metrics use the base_aligned/ column prefix.

• Candidate preparation and reranking evaluation
• Grep planning and execution
• Route comparison
• Label-coordinate audit

The same checkout contains an earlier top-50 comparison using the published coordinates and a separate embedding exploration with 15 queries from this repository. Those are different experiments; their figures are not mixed into the primary tables here.