How We Run CodeNib Locally on One DGX Spark

The artifact is the product boundary

A CodeNib manifest binds every published view to the repository identity, exact commit, filtered source fingerprint, and artifact hashes. The runtime checks that identity before exposing source. If the checkout changes without a compatible index update, CodeNib refuses to silently present old context as current.

repository@commit
        │
        ▼
incremental repository compiler
        ├── BM25 and regex/trigram search
        ├── dense vectors and fusion
        ├── source chunks
        └── symbol graph and SCIP/LSP navigation
        │
        ▼
verified capability manifest
        ├── source-linked Wiki and Ask
        ├── CodeGraph and Subsystem maps
        └── MCP context for coding agents

That separation lets us switch the generation route without rebuilding or weakening the repository evidence. A local Qwen server and a hosted DeepSeek profile can reuse the same manifest, cache, retrieval views, citations, and source verification.

Our pinned local reference profile

The current recorded profile is intentionally conservative. We cap context below the model maximum and keep both inference services on loopback so the machine has room for repository indexes, the web runtime, KV cache, and concurrent tooling.

Reference profile captured on one GB10 system. Pin exact model and container revisions before benchmarking. DGX Spark hardware details are published in NVIDIA's documentation; the generation checkpoint and serving guidance live in the Qwen model card.

Start generation and embeddings separately

We run two OpenAI-compatible endpoints. Generation receives most of the memory budget; embeddings use a small pooling runner. Both are bound to 127.0.0.1.

# Text-only generation on port 8080.
vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
  --revision 95a723d08a9490559dae23d0cff1d9466213d989 \
  --host 127.0.0.1 \
  --port 8080 \
  --served-model-name qwen3.6-35b \
  --max-model-len 65536 \
  --gpu-memory-utilization 0.60 \
  --max-num-seqs 2 \
  --language-model-only \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":1}'

# Dense embeddings on port 8081.
vllm serve Qwen/Qwen3-Embedding-0.6B \
  --revision 97b0c614be4d77ee51c0cef4e5f07c00f9eb65b3 \
  --host 127.0.0.1 \
  --port 8081 \
  --served-model-name Qwen/Qwen3-Embedding-0.6B \
  --runner pooling \
  --gpu-memory-utilization 0.08 \
  --enforce-eager \
  --no-enable-prefix-caching \
  --seed 0

The one-token MTP setting is an optimization, not a CodeNib requirement. Remove it for an autoregressive baseline or when the selected vLLM build does not support Qwen MTP. Verify both services before starting CodeNib:

curl -fsS http://127.0.0.1:8080/v1/models | jq .
curl -fsS http://127.0.0.1:8081/v1/models | jq .

Compile once, then add the consumers you need

Install the graph, MCP, model-backed Wiki, and OpenAI-compatible embedding clients. The embedding process is local, but CodeNib talks to it through the remote-compatible protocol.

python -m pip install \
  "codenib[agent,graph,mcp,semantic-remote]==0.2.1"

export REPOSITORY=/absolute/path/to/repository

# Model-free BM25, source, graph, and MCP onboarding.
codenib codegraph init "$REPOSITORY"
codenib codegraph status "$REPOSITORY"

# Add the optional dense view and generated Wiki.
export CODENIB_LOCAL_API_KEY=local-only
codenib wiki "$REPOSITORY" \
  --preset semantic \
  --generate \
  --model openai/qwen3.6-35b \
  --api-base http://127.0.0.1:8080/v1 \
  --api-key-env CODENIB_LOCAL_API_KEY \
  --embedding-provider openai \
  --embedding-model Qwen/Qwen3-Embedding-0.6B \
  --embedding-dimension 1024 \
  --embedding-endpoint http://127.0.0.1:8081/v1 \
  --embedding-api-key-env CODENIB_LOCAL_API_KEY

The semantic update preserves an independently current graph view. Later runs reuse current artifacts and update only the affected views when the source changes.

Keep local and hosted generation as thin overlays

The multi-repository demo uses layered YAML instead of copying one large configuration for every inference backend:

qa_config.yaml                 shared repositories and retrieval defaults
└── qa_config.local.yaml       machine paths, embeddings, local Qwen route
    └── qa_config.api.yaml     hosted generation override only

Mappings merge recursively, scalar values override their parents, and environment variables override the final YAML. An explicit null clears an inherited endpoint or credential. The local and API files are ignored by Git so machine paths and credentials do not enter a commit.

cp qa_config.local.yaml.example qa_config.local.yaml
cp qa_config.api.yaml.example qa_config.api.yaml

# Local Qwen profile.
export CODENIB_DEMO_MODEL=openai/qwen3.6-35b
export CODENIB_DEMO_API_BASE=http://127.0.0.1:8080/v1
export CODENIB_DEMO_API_KEY=local-only
bash scripts/start_web.sh

# Hosted generation, same indexes and Wiki cache.
export CODENIB_DEMO_API_KEY="$DEEPSEEK_API_KEY"
CODENIB_DEMO_PROFILE=api bash scripts/start_web.sh

Cold loading, retrieval, and generation are different waits

“Local” does not mean every first request is instantaneous. A useful latency report separates repository authorization and view loading, retrieval, and uncached prose generation.

In one August 2026 acceptance smoke across 26 repositories, warm dense queries took 11–25 ms. Generated Overview pages already on disk returned in roughly 3–4 ms on their first read after a backend restart and around 0.7 ms warm. Full repository authorization and view loading ranged from about 0.5 seconds for Requests and bat to 16.2 seconds for Babel. In an earlier three-turn smoke, three fixed local Qwen3.6 Ask cases took 19.6–23.5 seconds end to end.

These are operational observations from one host, not a model leaderboard. The generation sample is too small for a quality or throughput claim. A publishable comparison also needs model and runtime revisions, context length, cache state, TTFT, output tokens per second, and request concurrency.

Qwen3.8 plus DFlash2 versus the A3B baseline

We have now run the comparison on the same DGX Spark. It is important to name the systems precisely because this is not a same-model engine benchmark. Qwen/Qwen3.6-35B-A3B-FP8 is a mixture-of-experts model with 35 billion total parameters and about 3 billion activated per token; we serve it with vLLM and one built-in MTP draft token. Qwen/Qwen3.8-27B-FP8 is a dense 27-billion-parameter target. incoai/Qwen3.8-27B-DFlash2 is a five-layer draft model, not a standalone replacement: we configured an eight-position speculation block and the 27B target verifies its proposals.

Our pins were Qwen3.6 revision 95a723d08a9490559dae23d0cff1d9466213d989 on vLLM 0.19.2rc1.dev134+gfe9c3d6c5; Qwen3.8 target revision 017b9c7af6b5689d5dd426a76e0bc077eb5ca20a; DFlash2 draft revision dedf8df68adfb1afeaf7b7480c0a0243108177b4; and SGLang commit c14312a, which contains the merged DFlash2 path. The tested alternative endpoint was:

python -m sglang.launch_server \
  --model-path Qwen/Qwen3.8-27B-FP8 \
  --served-model-name qwen3.8-27b-dflash2 \
  --host 127.0.0.1 \
  --port 8082 \
  --context-length 65536 \
  --mem-fraction-static 0.65 \
  --max-running-requests 1 \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --speculative-algorithm DFLASH \
  --speculative-draft-model-path incoai/Qwen3.8-27B-DFlash2 \
  --speculative-num-draft-tokens 8 \
  --disable-overlap-schedule

Direct decode

Both systems used the same host, loaded local embedding service, 65,536-token context cap, request concurrency of one, temperature zero, disabled thinking, and 256 output tokens. These are warm steady-state client measurements; model download time is excluded.

DFlash2 works: compared with ordinary decoding of the same Qwen3.8 target at about 7.9 tok/s, it accelerated these prompts by 2.95–6.17×. But the cross-model product decision has a different winner. The A3B model's much smaller active parameter path was still 1.42–2.67× faster in decode and 1.41–2.63× lower in end-to-end latency than Qwen3.8 plus DFlash2 on this GB10.

Acceptance was workload-dependent. DFlash2 accepted roughly 39–40% of proposed tokens on the code and prose prompts and 76% on the integer sequence. A highly repetitive warm sequence reached 99%. This is why an advertised accelerator speedup cannot be treated as one fixed number.

Source-linked Ask

Raw decode is only one part of CodeNib latency. We also ran the same fixed Requests, Gin, and Vue questions through the complete retrieval, tool-call, evidence-review, and citation path. Both models received five turns and used the same repository artifacts and embedding endpoint.

The A3B path is the clear latency choice. The small quality sample is less one-sided: both models named every required symbol and returned valid source ranges, while Qwen3.8 retrieved more of the expected files. It also made 5–7 retrieval calls per case, versus five for A3B, which partly explains both the extra evidence and the longer tail.

We manually reviewed a fourth, adversarial Requests question about cookie precedence. A shallow reading of merge_cookies() suggests that session cookies win because one branch passes overwrite=False. In the actual Session.prepare_request() path, the request dictionary has already become a CookieJar; RequestsCookieJar.update() therefore overwrites a cookie with the same name, domain, and path. Qwen3.8 used its fifth turn to retrieve that method and corrected the answer. Qwen3.6 still made the shallow precedence claim with the same five-turn budget. This is useful evidence for retrieval behavior, not a general quality ranking: three fixed questions and one trap are far too small for that.

The code and sequence outputs from DFlash2 matched our ordinary Qwen3.8 hashes; the prose output did not. The target still verifies proposed tokens, but a different numerical kernel path can move a greedy boundary. We therefore do not describe this GB10 stack as byte-identical without a broader conformance test.

Warm cached startup was about five minutes for A3B and 6.4–7.0 minutes for DFlash2, where most of the latter was target and draft loading plus prefill CUDA-graph capture. Both coexisted with the embedding server and completed the tests without an OOM or container restart. Only concurrency one was validated.

Local and hosted are deployment profiles

Our public demo currently uses DeepSeek for user-facing generation to keep interactive latency predictable. The recorded reference profile uses local Qwen generation and local embeddings. Both retain the same source-bound repository architecture; neither changes what CodeNib considers verified context.

The product principle stays simple: index a codebase once, explore it as a Wiki, search it as context, and serve it to any coding agent. Start with the installation guide, follow the local and hosted profile guide, or inspect the source on GitHub.