When should a local RAG agent retrieve? A six-case walkthrough

By Vaibhav Singh, a coauthor of ThinkRetrieve. Published September 6, 2026. Prepared with AI assistance from the project documentation and saved experiment records.

A retrieval query can change after a model starts reasoning. That is the practical idea behind ThinkRetrieve: ask for an interim answer, use it together with the original question to retrieve a worked solution, and then let the model continue with that example.

This post walks through a small local-model demonstration. It uses six paraphrases of one synthetic incident, selected after exploratory screening. These are development examples, not a held-out benchmark.

The task: remember the right procedure

The simulated agent receives observations about a laboratory incident. It must infer a cause, retrieve one of four private procedures, and choose an opaque tool route such as dispatch_route("atlas_blue"). The names have no useful meaning without the procedure bank. No real laboratory action is executed.

The raw incident wording resembles one procedure, while a correct interim diagnosis points to another. This makes it possible to inspect whether retrieving after diagnosis changes the selected memory.

observations → interim diagnosis → worked example → final tool selection

This is a demonstration of procedural retrieval, not laboratory advice.

What the recorded runs show

Both models ran locally through Ollama on an RTX 4080 Super, with temperature 0 and a configured 768-token thinking budget. The same four-entry bank was available to the retrieval conditions. Plain reasoning and extra thinking had no private-memory retrieval.

In the missed 9B example, the interim diagnosis was wrong and retrieval selected the wrong procedure. The mechanism can reinforce an incorrect hypothesis. It helps only when the evolving reasoning supplies a useful retrieval query and the bank contains an appropriate example.

The plain condition also lacks the private mapping needed for reliable tool selection. Its score should not be read as a general measure of the model’s reasoning ability.

Mean recorded latency was 6.06 seconds for ThinkRetrieve 4B and 8.36 seconds for ThinkRetrieve 9B, versus 6.15 and 7.65 seconds for plain reasoning. These single-machine timings are illustrative. Equal configured thinking budgets do not establish equal total computation: interim probes, replayed input, retrieval, and final generation also cost work.

Try the experiment locally

With Python, Git, and a running Ollama server:

git clone https://github.com/itsvaibhav01/ThinkRetrieve.git
cd ThinkRetrieve
python -m venv .venv
. .venv/bin/activate
python -m pip install -e './thinkretrieve[faiss]'
ollama pull qwen3.5:4b
python thinkretrieve/benchmarks/science_agent/run.py \
  --model qwen3.5:4b --output science_agent_results.json

The first run downloads the embedding model. Model downloads require network access and sufficient memory. For a free installation check without a model or API key, run python thinkretrieve/examples/offline_dummy.py; that example uses scripted responses and is not an accuracy test.

The experiment runner and 4B / 9B records used for this write-up are pinned to a source revision. The run command above checks out the current default branch; record your commit and model digest when comparing results.

A useful next experiment

Try this pattern on a held-out collection of solved support tickets or debugging runbooks. Freeze the evaluation cases first, remove overlap with the bank, and retain both successful and failed runs. Compare raw-question retrieval and interim-answer retrieval. To isolate timing itself, also compare the same retrieved example inserted before versus during reasoning.

Report complete outputs, retrieval choices, latency, and all model-call token usage. A larger test may show a smaller gain or no gain; this small example establishes a behavior worth testing, not a general superiority claim.

Read the longer walkthrough · Explore the recorded demo · Code

The separate research paper is ThinkRetrieve: Retrieval-Augmented Reasoning Traces for Test-Time Scaling, Singh et al. (2026), accepted to Findings of EMNLP 2026. The linked version is the arXiv preprint. If you use the method in research, the repository provides citation metadata.