Use case
Agents are token-hungry in a way chatbots aren't. One user instruction fans out into many model calls — planning, tool selection, reading tool output, reflection loops, retries. The cost compounds fast, and most of those calls are mechanical enough that a capable open-source model handles them fine. InferAll gives your agent framework one OpenAI-compatible base URL with a free OSS allowance behind it. Choose a free route for steps it handles well and a paid model when your evaluation justifies it. The first example keeps one route with exact mode; daily request quotas still apply.
A chat app is roughly one model call per message. An agent is the opposite: a single “handle this for me” can become a plan step, a tool-selection step, a step that reads and interprets the tool's output, a reflection step that decides whether the result is good enough, a retry when it isn't, and a final synthesis step. That's many round-trips per task, and every one of them spends prompt and completion tokens.
During development and testing the multiplier is worse, because you re-run the same loops repeatedly while you tune prompts and tools. The tokens you spend proving the loop works dwarf the tokens any single production run costs. This is precisely where a free allowance earns its keep — the iteration phase is high-volume and low-stakes, the two properties that make a free OSS model the right call.
InferAll catalogs 10+ open-source models hosted on NVIDIA NIM — all billed at $0 input/$0 output against your starter balance. Activation is the $5 starter pack; that $5 covers your spendable balance and unlocks paid providers when you want them. The open NIM calls don't draw from your balance, so the $5 stretches as far as your daily request quota allows.
For an agent framework that supports an OpenAI-compatible chat endpoint, set the OpenAI base_url to https://api.inferall.ai/v1, use your InferAll key (prefix ifu_) as the API key, and set an explicit model ID. InferAll handles provider translation. Framework-specific features still need testing; base URL compatibility is not a guarantee for every workflow.
Sign in at /keys to create a key. Trial-eligible accounts can try 25 free NIM calls before payment. After the trial, complete checkout at /billing for the $5 starter pack, which becomes spendable balance. The gateway also speaks the Anthropic format at /v1/messages and exposes a native endpoint at /ai/v1/generate, but for agent frameworks the OpenAI-compatible /v1 base URL is the one you want.
OpenAI SDK (JavaScript) — one local tool round trip, exact routing
// Save as agent.mjs. Install: npm install openai
// Set INFERALL_API_KEY in your environment, then run: node agent.mjs
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.inferall.ai/v1",
apiKey: process.env.INFERALL_API_KEY,
defaultHeaders: { "X-Inferall-Routing": "exact" },
maxRetries: 0, // Keep this first test bounded; SDK retries are separate.
timeout: 60_000,
});
const model = "nvidia/nemotron-3-ultra-550b-a55b";
const messages = [{ role: "user", content:
"Use list_issues to get the open issues, then summarize them." }];
const tools = [{ type: "function", function: {
name: "list_issues",
description: "Read the local sample of open repository issues.",
parameters: { type: "object", properties: {}, additionalProperties: false },
} }];
const { data: first, response } = await client.chat.completions.create({
model, messages, tools, max_tokens: 1024,
}).withResponse();
console.log("Actual route:", response.headers.get("x-inferall-actual-model"));
console.log("Fallback:", response.headers.get("x-inferall-fallback-used"));
const choice = first.choices[0];
const calls = choice?.message?.tool_calls ?? [];
if (choice?.finish_reason !== "tool_calls" || calls.length !== 1) {
throw new Error("Expected one complete tool call; no tool was executed.");
}
const call = calls[0];
if (call.type !== "function" || call.function.name !== "list_issues") {
throw new Error("Unexpected tool; nothing executed.");
}
const args = JSON.parse(call.function.arguments);
if (!args || Array.isArray(args) || typeof args !== "object" || Object.keys(args).length) {
throw new Error("Expected empty tool arguments; nothing executed.");
}
// Local fixture only: replace with your own authorized read-only tool later.
const issues = [{ id: 1, title: "Add a retry-limit setting" },
{ id: 2, title: "Document timeout handling" }];
messages.push(choice.message);
messages.push({ role: "tool", tool_call_id: call.id, content: JSON.stringify(issues) });
const final = await client.chat.completions.create({ model, messages, max_tokens: 1024 });
if (final.choices[0]?.finish_reason !== "stop" || !final.choices[0]?.message?.content) {
throw new Error("No complete summary returned; inspect the response before continuing.");
}
console.log(final.choices[0].message.content);
// Check that the summary matches the two fixture issues. Two API calls are used.
// For a paid comparison, explicitly select anthropic/claude-sonnet-4-6.
// That requires paid access and uses InferAll's published model rates.The core move is to route by step class, not to pick one model for the whole agent. An open-source model and a frontier model have different sweet spots, and an agent conveniently separates into steps that map onto each.
Start with a bounded task: ask a free model to call a local read-only tool, send the result back, and inspect the summary. The example above validates the tool call before executing it and stops after two model requests. It uses exact routing so an alternative model cannot change your evaluation.
The hard steps — decompose an ambiguous goal into an ordered plan, reason over a large heterogeneous context, recover from a failed multi-step sequence — are where you want a frontier model. Set the model to anthropic/claude-sonnet-4-6 or openai/gpt-4o for just that call, then drop back to the open NIM model for the rest of the loop. Because the hard steps are the rare ones, you pay frontier rates on a small slice of traffic while the $0-rate open lane absorbs the bulk.
For the heavier planning calls you can also reach for anthropic/claude-opus-4-8 or a long-context Gemini model like gemini/gemini-2.5-pro (or gemini/gemini-2.5-flash when you want a cheaper frontier option). All of them are explicit model choices through the same key and base URL. Paid access and each model's context limits still apply; the gateway does not automatically select a larger context window for your task.
LangChain — two tiers, one gateway
# LangChain: same gateway, one ChatOpenAI per "tier".
# Use the free model for the bulk of the loop and a frontier model
# only for a step where your evaluation shows an improvement.
# This example uses default auto routing: fallback may change the model.
# For exact routing, your framework must forward X-Inferall-Routing: exact.
# The OpenAI SDK example above shows a verified header configuration.
from langchain_openai import ChatOpenAI
# Free token rate; trial access and daily request quotas still apply.
cheap = ChatOpenAI(
base_url="https://api.inferall.ai/v1",
api_key="ifu_...",
model="nvidia/nemotron-3-ultra-550b-a55b", # check current catalog pricing
)
# Paid initial route for the planning call.
planner = ChatOpenAI(
base_url="https://api.inferall.ai/v1",
api_key="ifu_...",
model="anthropic/claude-sonnet-4-6", # billed at InferAll's published rate
)
# Route by step. Most calls hit `cheap`; `planner` is the rare one.
plan = planner.invoke("Decompose this goal into ordered steps: ...")
step = cheap.invoke(f"Summarize the first step of this plan: {plan.content}")
print(step.content)The OpenAI-compatible endpoint accepts function tool schemas. Model support and argument quality vary, so validate tool names and arguments in your application. Then execute only the tools you intended to expose and return the result with its tool-call ID. The model proposes a call; the gateway does not run your tool.
Start with the local fixture above before connecting external services or enabling writes. A successful text response does not demonstrate a working tool loop. Check that the final answer uses the actual tool result, and stop on an incomplete response.
InferAll catalogs 10+ open-source models hosted on NVIDIA NIM. Check the catalog for current token rates and availability. Examples to evaluate:
The roster shifts as NIM adds and retires models, so don't hardcode a list from memory. Fetch GET https://api.inferall.ai/ai/v1/models for the live catalog and which IDs are currently $0, or browse the docs.
With X-Inferall-Routing: auto, or no routing header, a recoverable upstream failure can trigger another eligible model or provider. Eligible free chat routes can also run a parallel backup request. Free-model requests only fall back to other eligible free models, which may be different models. Paid requests may change provider. A prefix selects the initial route, not a guarantee that the same model answers.
Send X-Inferall-Routing: exact to disable alternative routes. The SDK example above sets it on every request using defaultHeaders. If that route fails, you receive the failure. Read X-Inferall-Actual-Model, X-Inferall-Actual-Provider and X-Inferall-Fallback-Used when present; validation or access errors may happen before any provider is contacted.
Exact routing keeps the selected public model identity, but a provider-owned alias can still change upstream. InferAll refuses an alias in exact mode if it would rewrite it to another ID. Context limits, access rules and quotas still apply. Shorten an oversized prompt or choose another supported model explicitly; there is no promised automatic 128k-to-Gemini upgrade. Bound client retries and loop turns separately from routing mode.
We won't invent dollar figures — they shift with model rates and with how your agent fans out. The structural argument is what matters: an agent splits into many cheap calls and a few expensive ones, and you only pay frontier rates on the few.
If roughly ten of every twelve calls in a task are simple enough for an open NIM model, those ten run at $0 input/ $0 output (subject to your tier's daily request quota), while the two hard calls go to a pinned frontier model at InferAll's published rate. The development and testing phase, where you re-run loops constantly, is where the $0-rate open lane does the most work. The exact savings depend on your call fan-out and how often you stay under the allowance — run it against your own numbers.
Worked example
# Illustrative cost shape for an agent — not a guarantee, not a promise.
# Substitute your per-step token counts and InferAll's published rates.
# Why agents are different: ONE user task fans out into many calls.
# plan -> pick tool -> read tool output -> reflect -> retry -> answer
# Say a single task averages ~12 model calls.
# A hypothetical day of development / testing:
# 200 agent tasks x ~12 calls each = ~2,400 calls
# Of those calls:
# ~10 calls/task are simple (classify, summarize, loop control)
# ~2 calls/task are hard (plan, reason over big context)
# Through InferAll:
# Use exact routing if this comparison must keep the selected models.
# Simple calls -> open NIM (nvidia/nemotron-3-ultra-550b-a55b):
# billed at $0 input / $0 output against your $5 starter balance
# (subject to your tier's daily request count quota)
# Hard calls -> anthropic/claude-sonnet-4-6 or openai/gpt-4o:
# billed at our published per-token rates.
#
# Net: you pay frontier rates on the ~2-of-12 calls that need a
# frontier model, and $0 (within allowance) on the ~10-of-12 that
# don't. The development/test phase — where you re-run loops
# constantly — is where the free allowance does the most work.
# Run the numbers against your own call fan-out before assuming
# the magnitude.Free OSS, almost entirely. While you're tuning prompts and tools you re-run the loop constantly, and that iteration is high-volume and low-stakes — the exact shape the free allowance is built for. Pin a free model for the whole loop during development and only switch pieces to a frontier model once you've confirmed the step genuinely needs it.
Mix. Simple, well-described tools and short chains run on a free model. The orchestration step that picks tools from a large catalog and sequences them is where you pin anthropic/claude-sonnet-4-6 or openai/gpt-4o for that one call. Your application executes validated tools; a free model can then summarize their returned results.
Free OSS for the “is this good enough?” check and the retry; it's a frequent, mechanical judgment. Reserve a frontier model for the case where repeated retries keep failing and you need stronger reasoning to figure out why — a rare call relative to the loop's volume.
Frontier. Turning an ambiguous goal into a correct, ordered plan is the call most worth paying for, and it's usually one call near the start of a task. Pinning anthropic/claude-opus-4-8 or anthropic/claude-sonnet-4-6 here and free OSS for execution is the canonical split.
Be careful. An agent that calls the model on every step without a turn cap can burn the daily request quota quickly. Free-model fallback stays on eligible free routes; it does not bypass a quota by switching to a paid model. Calls you explicitly send to premium routes use their published rates and remain subject to access and limits. Cap turns and client retries, and watch the dashboard. The cost story for an unbounded agent is fundamentally different from an interactive session.
We built InferAll because we wanted exactly this for our own agents — one OpenAI-compatible base URL, a free open-source allowance for the high-volume inner-loop calls, and frontier models on tap for the planning steps, all behind one key and one bill. The alternatives were OSS gateways we'd have to operate, observability tools that didn't solve routing, or single-provider keys that locked us into one upstream. So this page is biased: we're describing the workflow we're selling. The technical claims should still hold up if you check them, and the corrections email at the bottom is real. If we got something wrong, tell us.
Why can free token rates help an agent workload?
An agent task can make several model calls for planning, tool selection, reading results and retries. Using a model listed at $0 input and $0 output can reduce token spend for steps it handles well. It does not remove daily request quotas or guarantee that the result is useful. Measure completed tasks, latency and retries on your own workload; see /models for rates and /pricing for current plan limits.
How do I point an agent framework at the gateway?
Use https://api.inferall.ai/v1 as the OpenAI-compatible base URL, your InferAll API key, and an explicit model ID such as nvidia/nemotron-3-ultra-550b-a55b. The JavaScript OpenAI SDK example below also sets defaultHeaders to send X-Inferall-Routing: exact, disabling alternative-model/provider fallback. LangChain's ChatOpenAI accepts base_url, api_key and model, but those three settings alone keep default automatic fallback enabled. Check your framework version's custom-header support when you need exact routing.
Can free models do tool calling?
The chat endpoint accepts function tool schemas, but support and reliability depend on the model and workflow. Your application must validate a returned tool name and arguments, execute the tool, and send its result back. The example below performs that round trip against a small local issue fixture; it does not access a repository service. A text answer alone does not prove tool execution, and an interrupted stream is not a completed tool call.
How do I choose free and paid models for different steps?
Evaluate a free model for summarizing or classifying tool results, then explicitly choose a paid model for a step that benefits from it. Use provider-prefixed IDs such as anthropic/claude-sonnet-4-6 or openai/gpt-4o. Trial accounts cannot use paid routes; accounts that have paid can also use supported bare paid-model IDs. Exact routing keeps the selected route or returns an error. Token prices are InferAll's published rates, which may differ from upstream rates.
Can fallback change my model or provider?
Yes. With X-Inferall-Routing omitted or set to auto, recoverable failures can trigger another eligible model or provider; eligible free chat routes can also use parallel backup requests. A request for a free model falls back only to other eligible free models, not necessarily the same model. Paid requests may change provider. Send X-Inferall-Routing: exact to disable alternative routes and receive a failure if the selected route cannot answer. A model prefix alone does not disable fallback. Exact mode does not bypass billing, quotas or context limits, and provider-owned aliases can still change upstream.
How does pricing work?
Trial-eligible new accounts get 25 free NIM calls before payment. After the trial, the $5 starter pack unlocks ongoing access and becomes spendable balance. Models priced at $0 input and output do not draw from it; daily request quotas still apply. Premium text models use the published rates on /models. See /pricing for current plan limits. Bound your agent's turns, output tokens and client retries; exact routing is not a spending limit.
What should I do if a prompt exceeds the model context limit?
Shorten the prompt or explicitly choose a supported model with a suitable context window. Do not rely on an automatic 128k-to-Gemini upgrade. An explicit provider-prefixed model selects that initial route, and exact mode disables alternatives. Invalid or unsupported IDs are refused rather than silently mapped to a free model; exact mode also refuses model aliases that InferAll would rewrite to a different ID.
Where can I find current free model IDs?
The example uses nvidia/nemotron-3-ultra-550b-a55b. Check GET https://api.inferall.ai/ai/v1/models or /models for the current catalog and token prices. openai/gpt-oss-20b is a NIM-hosted open-weight model despite its openai namespace. A catalog listing does not prove provider uptime, tool support or latency; test your selected route with a bounded task.
Claude Code on free OSS models — the same route-by-task pattern, applied to Claude Code.
InferAll home — the gateway, the $5 starter, the failover story.
InferAll for VS Code — Cline-based agent with the gateway pre-wired, free first run.
Rate limits — free-tier caps, per-key behavior, what happens on 429.
Pricing — current plans and request quotas.
AI inference API — endpoint surface, supported providers, code examples.
Unified AI API — one key, one bill, every provider.
Last updated: 2026-10-07.
InferAll facts on this page are drawn from this site and the gateway running at api.inferall.ai. Model availability on NVIDIA NIM may shift as upstream hosts change their roster. Specifics change. Have a correction? Email contact@kindly.fyi.