Docs

Use InferAll with Hindsight

Hindsight is an open-source memory server for AI agents. It uses an LLM to extract facts when it stores memories and to answer from them. Point its OpenAI provider at InferAll and it runs on a free NVIDIA model with no other change.

Tested on 2026-10-28 with the official Docker image (hindsight-api 1.1.0). For Hindsight's own options, see its model docs.

1. Set your key

Create a key at /keys and export it in the terminal you will use. The container reads it from the environment, so it is not written into the command.

Shell

# Use your own key from inferall.ai/keys. Never paste it into a file you commit.
export HINDSIGHT_API_LLM_API_KEY="paste-your-inferall-key-here"

2. Start Hindsight

The image is about 3.6 GB and includes its database and a local embedding model, so only the LLM calls go to InferAll.

Docker

docker run -d --name hindsight -p 8888:8888 -p 9999:9999 \
  -e HINDSIGHT_API_LLM_PROVIDER=openai \
  -e HINDSIGHT_API_LLM_BASE_URL=https://api.inferall.ai/v1 \
  -e HINDSIGHT_API_LLM_API_KEY \
  -e HINDSIGHT_API_LLM_MODEL=nvidia/nemotron-3.5-super-vl-120b-a12b \
  -e 'HINDSIGHT_API_LLM_DEFAULT_HEADERS={"X-Inferall-Routing":"exact"}' \
  -v hindsight-data:/home/hindsight/.pg0 \
  ghcr.io/vectorize-io/hindsight:latest

# Wait until it answers
until curl -sf http://localhost:8888/health >/dev/null; do sleep 5; done

HINDSIGHT_API_LLM_BASE_URL ends in /v1; Hindsight appends /chat/completions.
nvidia/nemotron-3.5-super-vl-120b-a12b is $0 input and output. Paid ids from /models work the same way with paid access.
Avoid nvidia/nemotron-3-ultra-550b-a55b here. Hindsight forces its final done tool call, and Ultra then ends the answer at its first line break, so reflect returns one sentence. We measured that in all 3 of our reflect runs.
The X-Inferall-Routing: exact header keeps every call on that model. Without it, a slow call can be answered by another free model, which may be Ultra.

3. Store and ask

Retain, recall, reflect

B=http://localhost:8888/v1/default/banks/my-bank

# Retain: extract facts with the LLM and store them
curl -s -X POST $B/memories -H 'Content-Type: application/json' \
  -d '{"items":[{"content":"Alice is a backend engineer at Acme. She prefers Go over Python and is migrating the billing service to Postgres 17 this quarter."}],"async":false}'

# Recall: search stored memories (no LLM call)
curl -s -X POST $B/memories/recall -H 'Content-Type: application/json' \
  -d '{"query":"What is Alice working on?"}'

# Reflect: answer from memory with the LLM
curl -s -X POST $B/reflect -H 'Content-Type: application/json' \
  -d '{"query":"What should I know before talking to Alice about databases?"}'

In our two runs the three commands took 70-73s in total. Recall returned the three facts Hindsight extracted (Alice's role, her preference for Go, and the Postgres 17 migration this quarter), and reflect answered in 2,000+ characters of markdown. Hindsight retries a call that fails; 2 of 14 did, and both runs still completed.

That one retain and one reflect made 7 LLM calls through InferAll. A free trial key has 25 calls, then 25 a day, so a busy memory bank needs the $5 starter pack (50 a day) or a plan.