Documentation

Quick start

# Two lines, no install. Point any OpenAI-compatible client at InferAll:
#   base URL  https://api.inferall.ai/v1
#   api key   your ifu_... key from inferall.ai/keys

# Python SDK
pip install inferall-ai

# TypeScript SDK
npm install @inferall/sdk

# There is also an install wizard that edits your project for you:
npx @inferall/cli init

Want a runnable example? examples/ in the repo has minimal Python and TypeScript scripts you can clone, edit, and run.

Base URL

https://api.inferall.ai

All requests require an API key via Authorization: Bearer ifu_... or x-api-key: ifu_... (legacy kr_proj_ keys are still accepted).

Endpoints

MethodPathDescription
POST/ai/v1/generateGenerate text, chat, images, or video
GET/ai/v1/modelsList all models with pricing
GET/ai/v1/healthHealth check
POST/v1/chat/completionsOpenAI Chat Completions, every model
POST/v1/responsesOpenAI Responses API (OpenAI SDK, Agents SDK)
GET/v1/modelsOpenAI-style model list (ids to send as model, with pricing per token)
POST/v1/messagesAnthropic-compatible (Claude Code)
POST/v1/messages/count_tokensAnthropic token counting (exact for Claude, estimate otherwise)
POST/v1/systemoneSystem One (JEV) — typed decisions, not chat
POST/ai/v1/keysCreate API key (requires JWT)
GET/ai/v1/keysList your keys (requires JWT)
GET/ai/v1/usageUsage summary (requires JWT)
GET/ai/v1/requestsYour request history (requires JWT)
GET/ai/v1/key/statusTrial counter, balance, savings — auth via API key
POST/ai/v1/billing/checkoutStripe checkout session
GET/ai/v1/billing/statusBilling status and spend

Choose model routing

Set X-Inferall-Routing on each request to /ai/v1/generate, /v1/chat/completions, /v1/messages or /v1/responses. Use auto, the default when omitted, when availability matters: the gateway may try another eligible model after a failure or run a parallel backup request for eligible free models. Use exact for tests or workflows that depend on a particular model: it disables automatic alternative-model/provider fallback and parallel backup requests. If that route fails, you receive the failure.

A provider prefix such as openai/gpt-4.1 selects the initial route; exact keeps that public model or service identity. Provider prefixes, NVIDIA namespaces and eligible bare paid-model names still resolve when they identify the same model. Unsupported or ambiguous IDs are refused. If InferAll would replace a retired model or remove a date suffix to use another model ID, exact returns HTTP 400 with exact_model_unavailable. In auto, existing replacements can still occur; check X-Inferall-Actual-Model. Fixed audio services on Generate also require their published model ID in exact mode.

An alias passed unchanged to its provider can still move to a newer version. Use a supported versioned model ID when you need a specific version; exact does not guarantee that a historical version remains available. Trial access and billing rules still apply. Values are case-insensitive; an empty value or a value other than auto or exact returns HTTP 400.

# Set INFERALL_API_KEY to your InferAll key. This example uses a paid model.
# -i prints the routing headers as well as the response body.
curl -i https://api.inferall.ai/v1/chat/completions \
  -H "Authorization: Bearer $INFERALL_API_KEY" \
  -H "Content-Type: application/json" \
  -H "X-Inferall-Routing: exact" \
  -d '{
    "model": "openai/gpt-4.1",
    "messages": [{"role": "user", "content": "Explain gravity in one sentence."}],
    "max_tokens": 100
  }'

Read the response headers to compare what you requested with the route used. They are available for streamed and buffered responses, and browser clients can read them through CORS. With an SDK, use its raw-response access to inspect headers. Non-ASCII and control characters in model header values are percent-encoded as UTF-8 bytes.

X-Inferall-Routing
The request's routing mode: auto or exact.
X-Inferall-Requested-Model
The model requested by the caller.
X-Inferall-Original-Provider
The provider selected for the initial route.
X-Inferall-Actual-Provider / X-Inferall-Actual-Model
The provider and resolved public model ID for the attempt. InferAll-branded models use the public provider name inferall.
X-Inferall-Fallback-Used / X-Inferall-Fallback-Reason
Fallback-Used is true or false; the reason is present only when fallback was used.

Routing headers can also describe an attempted route that returned a provider error. Check the response status and body for the outcome; these headers identify the selected attempt and do not prove provider receipt, task completion or the amount charged. Early authentication, validation and billing refusals omit an actual provider/model identity.

Compatibility

The gateway translates OpenAI-style and Anthropic-style requests to each provider's own format. Capabilities vary by model, provider and endpoint, so check the model you choose and the endpoint you call rather than assuming every feature is available everywhere. The OpenAI and Anthropic SDKs, the OpenAI Agents SDK, the Vercel AI SDK, LangChain and Claude Code can be pointed at InferAll.

Not supported yet: previous_response_id and OpenAI's hosted tools (web search, file search) on /v1/responses; these return a 400 that names the parameter. Guides: Pi setup guide, OpenCode setup guide, OpenAI Agents SDK, Vercel AI SDK, Claude Desktop.

Structured output

On /v1/chat/completions, send response_format with json_schema; on /v1/responses, send the same schema in text.format. Setting strict: true explicitly asks for strict handling, and the gateway preserves it on both endpoints. The error cases below describe requests with X-Inferall-Routing: exact; when automatic fallback is enabled, a failed request can return the initial provider's error.

Each provider supports its own subset of JSON Schema, and the subsets differ. An unsupported_json_schema error means the selected path cannot preserve the requested constraints. Pick a schema the provider supports or a different model; if best-effort output is acceptable for your use, choose non-strict explicitly.

A catalog listing alone does not establish structured-output support. Validate the final output against your schema in your application, and check the values themselves (for example, that blocked_count matches the tickets). A valid shape can still hold a wrong answer.

The example below uses the official OpenAI SDK (7.30.0) and Zod (4.6.5) with zodResponseFormat, which sends strict: true. It runs on openai/gpt-4.1, a paid model, so each run spends InferAll credit; see pricing and billing. It sets X-Inferall-Routing: exact, maxRetries: 0 and max_tokens: 1024. For the synthetic prompt, a useful result has 2 tickets and blocked_count 1. Save the code as structured-output.ts, then run the setup commands.

# Uses a paid model (openai/gpt-4.1): this spends InferAll credit.
npm install openai@7.30.0 zod@4.6.5 tsx
export INFERALL_API_KEY=your_inferall_key
npx tsx structured-output.ts
// Structured output through InferAll with the official OpenAI SDK and Zod.
//
// This example uses a PAID model (openai/gpt-4.1) and spends InferAll credit.
//
// Setup:
//   npm install openai@7.30.0 zod@4.6.5 tsx
//   export INFERALL_API_KEY=your_inferall_key
//   npx tsx structured-output.ts

import OpenAI from "openai";
import { zodResponseFormat } from "openai/helpers/zod";
import { z } from "zod";

const Summary = z.object({
  tickets: z.array(
    z.object({
      id: z.string(),
      owner: z.string().nullable(),
      status: z.enum(["blocked", "done"]),
    }),
  ),
  blocked_count: z.number().int(),
});

const apiKey = process.env.INFERALL_API_KEY;
if (!apiKey) throw new Error("Set INFERALL_API_KEY to your InferAll key before running this example.");

const client = new OpenAI({
  baseURL: "https://api.inferall.ai/v1",
  apiKey,
  maxRetries: 0,
  timeout: 60_000,
  defaultHeaders: { "X-Inferall-Routing": "exact" },
});

async function main() {
  const completion = await client.chat.completions.parse({
    model: "openai/gpt-4.1",
    max_tokens: 1024,
    messages: [
      {
        role: "user",
        content:
          "Summarize these tickets. Ticket I-1 owner Mira status blocked; I-2 no owner status done.",
      },
    ],
    response_format: zodResponseFormat(Summary, "ticket_summary"),
  });

  const message = completion.choices[0]?.message;
  if (message?.refusal) {
    console.error("Model refused:", message.refusal);
    process.exit(1);
  }
  if (!message?.parsed) {
    console.error("No parsed output returned.");
    process.exit(1);
  }

  const summary = message.parsed;
  const actualBlocked = summary.tickets.filter(ticket => ticket.status === "blocked").length;
  if (summary.blocked_count !== actualBlocked) {
    throw new Error("The blocked count does not match the extracted tickets.");
  }
  // For this input, expect 2 tickets and blocked_count 1.
  console.log(JSON.stringify(summary, null, 2));
}

main().catch((err) => {
  console.error(err instanceof Error ? err.message : "The request failed.");
  process.exit(1);
});

Download structured-output.ts to run the example locally.

Request history

Open Usage to inspect requested and served models, providers, fallback reasons, response outcomes, latency, tokens, and estimated cost. Filter by API key or add project and run labels to your inference calls:

X-Inferall-Project: my-app
X-Inferall-Run: agent-run-42

These optional headers work on /ai/v1/generate, /v1/chat/completions, /v1/responses, and /v1/messages. Labels use 1–64 ASCII letters, digits, periods, underscores, colons, or hyphens; invalid labels are ignored. Labels organize history and do not change permissions or budgets. Look for X-Inferall-Request-Id in the response headers when matching a call to history or contacting support.

For programmatic access, GET /ai/v1/requests accepts a signed-in Supabase JWT in Authorization: Bearer, rather than an inference API key. Optional query parameters are from and to (ISO dates or timestamps; UTC), key_id, project, run, and limit (default 25, maximum 100). The default window is 30 days and each query can cover up to 90 days. Pass the returned next_cursor with the same filters to continue the original snapshot; refresh without a cursor to see newly logged events. Each page contains requests, next_cursor, and the period used.

New Chat Completions and Responses timeout events can include timeout_diagnostics. Expand the event in Usage to see whether the gateway was waiting for a provider response, initial stream output, or a response body, along with the routing mode and attempt timing. The phase records where it was waiting; it does not establish the cause. Missing diagnostics remain unknown, and recorded events may omit discarded attempts. The startup time limit applies to headers and, for OpenAI-format streams, initial output; it is not a response-body deadline.

History contains metadata, not prompts or generated content. It groups recorded events and may not include every attempt. Older records lack request IDs and labels, so key, project, and run filters exclude them. Missing values stay unknown. Costs are estimates, not invoice totals; a completed response does not establish that an agent finished its task.

Common errors

The gateway returns structured error messages that name the likely cause. Read the response body — the message field tells you exactly what to fix.

401 — wrong key prefix

If you point ANTHROPIC_BASE_URL/OPENAI_BASE_URL at InferAll but leave your existing provider key in place, the 401 message names the provider it came from:

  • • sk-... → “looks like an OpenAI API key”
  • • sk-ant-... → “looks like an Anthropic API key”
  • • sk-or-... → “looks like an OpenRouter API key”
  • • AIzaSy... → “looks like a Google AI Studio key”
  • • ifu_... but not in our DB → “key has an InferAll prefix but isn't in our database” (likely revoked or mistyped — mint a fresh one at /keys)

402 — billing required

Trial-eligible accounts (tier pending) get 25 free NIM-model calls before any payment. When those run out you will see a 402, and completing the $5 starter-pack checkout at /billing unlocks ongoing access to the 10+ open models priced at $0 in and out. The $5 becomes spendable balance, not a fee. Already activated and still seeing 402? Check that has_paid_successfully flipped after your charge cleared; if not, the webhook may have lagged, so retry once and file support if it persists.

429 / 529 — rate limited

Honor Retry-After when present, otherwise exponential backoff. See /docs/rate-limits for the per-tier daily caps and per-operation limits.

TypeScript SDK

The native TypeScript SDK is live on npm as @inferall/sdk 0.1.1 (npm install @inferall/sdk). Prefer the official OpenAI or Anthropic SDK? Point either at the InferAll base URL — both work unchanged, and npx @inferall/cli init makes that one-line change for you (and checks your setup any time with npx @inferall/cli doctor).

// Native TypeScript SDK — live on npm as @inferall/sdk 0.1.1
import { Inferall } from "@inferall/sdk";

const ai = new Inferall(); // reads INFERALL_API_KEY (ifu_...)

// Text. The SDK default is Gemini, which is a PAID model, so on the
// no-card trial pass a free NVIDIA model explicitly.
const text = await ai.text("Explain quantum computing in two sentences", {
  provider: "nvidia",
  model: "nvidia/nemotron-3-ultra-550b-a55b",
});

// Chat with a specific provider/model
const reply = await ai.chat(messages, {
  provider: "anthropic",
  model: "claude-sonnet-4-6",
});

// Vision
const analysis = await ai.vision(imageBase64, "What is this?");

// Image-to-video (Veo3) — pass a source frame for persona-consistency
const talkingHead = await ai.generate({
  provider: "gemini",
  model: "veo-3.1-generate-preview",
  operation: "video-generate",
  prompt: "Talking-head shot, looking at camera",
  source_image_url: "https://example.com/persona-portrait.jpg",
});

// Prefer the official OpenAI/Anthropic SDKs instead? Point them at InferAll:
//   OPENAI_BASE_URL=https://api.inferall.ai/v1   (key: ifu_...)
//   ANTHROPIC_BASE_URL=https://api.inferall.ai   (key: ifu_...)

Python SDK

from inferall import Inferall

ai = Inferall()  # reads INFERALL_API_KEY from the environment

# Text generation. The SDK default is Gemini, which is a PAID model, so on
# the no-card trial pass a free NVIDIA model explicitly.
text = ai.text(
    "Explain quantum computing in two sentences",
    provider="nvidia",
    model="nvidia/nemotron-3-ultra-550b-a55b",
)

# Chat with any provider
reply = ai.chat(messages, provider="anthropic", model="claude-sonnet-4-6")

# Vision
analysis = ai.vision(image_base64, "What is this?")

# Generate (image or video)
video = ai.generate(
    provider="gemini",
    model="veo-3.1-fast-generate-preview",
    operation="video-generate",
    prompt="Drone shot of a city",
)

# Image-to-video (Veo3) — pass a source frame to keep the subject consistent
talking_head = ai.generate(
    provider="gemini",
    model="veo-3.1-generate-preview",
    operation="video-generate",
    prompt="Talking-head shot, looking at camera",
    source_image_url="https://example.com/persona-portrait.jpg",
)

OpenAI SDK (drop-in)

Keep the code you already have. Point the OpenAI SDK at https://api.inferall.ai/v1 with your ifu_ key and the same call reaches any model we route. The models page marks the few we do not.

Read resp.model and the routing headers to see which model answered. An ID the gateway cannot route is refused with HTTP 400; it is not silently replaced with a free default. A supported request can still use automatic fallback. Set X-Inferall-Routing: exact when you need to keep the selected route; see model routing above.

# Already using the OpenAI SDK? Change two arguments.
from openai import OpenAI

client = OpenAI(
    base_url="https://api.inferall.ai/v1",
    api_key="ifu_your_inferall_key",
)

# Free NVIDIA NIM open models: $0 in, $0 out
resp = client.chat.completions.create(
    model="nvidia/nemotron-3-ultra-550b-a55b",
    messages=[{"role": "user", "content": "Explain quantum computing in two sentences"}],
)
print(resp.choices[0].message.content)

# Check what actually answered. Normally the id you sent; if that free model
# is overloaded, another $0 model answers and resp.model names it.
print(resp.model)

# Reach a premium provider with a model prefix, billed from your balance
resp = client.chat.completions.create(
    model="anthropic/claude-sonnet-4-6",
    messages=[{"role": "user", "content": "Same call, different provider"}],
)
print(resp.model)   # claude-sonnet-4-6, the prefix is routing and is stripped

# An ID the router cannot map is REFUSED with HTTP 400 naming that ID. It is not
# answered by a substitute, so there is no silent downgrade to watch for here.
# On a trial key, bare "gpt-4o" and bare "claude-sonnet-4-6" are refused. Prefix
# the ID with its provider and it routes: "openai/gpt-4o", "anthropic/claude-sonnet-4-6".
# Once your key has paid, bare catalogued claude-/gpt-/gemini- IDs route too.
#
# resp.model still names the model that answered, which is how you confirm a
# prefixed ID went where you meant.

Claude Code integration

Point Claude Code (or any Anthropic-SDK app) at InferAll with the two env vars below, and on the free trial, name a model this gateway can route: a trial account's bare Claude ID is refused. Once the account has credits, Claude Code's own Claude models work as-is, billed from your balance.

The free NVIDIA NIM open models are $0 input / $0 output and are what a new account can call, but you have to name one: this gateway refuses an ID it cannot map rather than picking a model for you. New accounts get 25 free NIM calls before any payment; after those, the $5 starter pack unlocks ongoing NIM access and that same $5 becomes spendable balance for premium providers via the anthropic/claude-sonnet-4-6 style prefix.

The same rule as the OpenAI SDK section above applies here, and it is the thing to know before you point a client at us. On /v1/messages a bare ID this gateway cannot map is refused with HTTP 400 naming that ID. It is not answered by a substitute, so claude-sonnet-4-6 sent bare on a trial account will fail rather than quietly route somewhere else. A prefixed ID such as anthropic/claude-sonnet-4-6 selects that provider; eligible paid accounts can also use supported bare Claude IDs. Routing headers show any fallback. Set X-Inferall-Routing: exact to disable alternative routes for a request.

# Use Claude Code with free inference
export ANTHROPIC_BASE_URL=https://api.inferall.ai
export ANTHROPIC_API_KEY=your_inferall_key

# ON THE FREE TRIAL: set the model too, or this will not be free.
# Claude Code sends a bare Claude ID such as claude-sonnet-4-6. On a trial
# account the gateway refuses it (HTTP 400 naming the ID); it never answers
# with a different model. Name a free model instead:
export ANTHROPIC_MODEL=nvidia/nemotron-3-ultra-550b-a55b
claude

# WITH CREDITS (any account that has bought the $5 starter pack): skip
# ANTHROPIC_MODEL. Claude Code's own Claude models route to real Anthropic,
# billed per token from your balance.

# The $0 models. These route as written and a no-card trial covers them:
#   nvidia/nemotron-3-ultra-550b-a55b    NVIDIA's 550B open model; it reasons before it answers
#   openai/gpt-oss-20b                   OpenAI's open-weight model, fast
#
# PAID, billed per token from your balance — prefix routes to the real provider:
#   anthropic/claude-sonnet-4-6  → actual Claude
#   gemini/gemini-2.5-flash      → Google Gemini

Prompt caching (Claude)

Mark a long, unchanging prefix (instructions, a document, tool definitions) and Claude reuses it on later calls until it goes five minutes unused. A cache read bills at 10% of the model's input rate and a cache write at 125%, both at the Claude rates on the models page. Anthropic caches only prompts above a model-specific minimum length; a shorter prompt is simply not cached.

On /v1/messages, send cache_control exactly as in Anthropic's API; Claude Code does this on its own. On /v1/chat/completions, put it on a text part of a system or user message, as below. Only the five-minute cache is available there; a ttl is ignored.

On /v1/chat/completions, prompt_tokens includes cached tokens, and prompt_tokens_details splits out cached_tokens (read) and cache_write_tokens (written) when the cache was used. /v1/messages returns Anthropic's own usage fields.

from openai import OpenAI

client = OpenAI(base_url="https://api.inferall.ai/v1", api_key="ifu_your_inferall_key")

resp = client.chat.completions.create(
    model="anthropic/claude-sonnet-4-6",
    messages=[
        {"role": "system", "content": [
            {"type": "text", "text": LONG_INSTRUCTIONS,
             "cache_control": {"type": "ephemeral"}},
        ]},
        {"role": "user", "content": "Summarize the latest ticket."},
    ],
)
# Present when the cache was used: cached_tokens (read), cache_write_tokens (written)
print(resp.usage.prompt_tokens_details)

Key status (trial counter + savings)

GET /ai/v1/key/status — auth via your ifu_/kr_user_ API key (Bearer or x-api-key). Returns the trial counter, balance, and 7-day savings vs paid providers — enough for an SDK or CLI to render an upgrade nudge inline.

{
  "tier": "pending",
  "project": false,
  "trial":   { "used": 50, "remaining": 150, "cap": 200 },
  "balance_cents": 500,
  "balance_usd": 5,
  "has_card": true,
  "has_paid_successfully": true,
  "savings_7d": {
    "vs_gpt4o_mini_usd": 0.0318,
    "vs_claude_sonnet_usd": 0.6373
  },
  "checkout_url": "https://inferall.ai/billing"
}

For per-call awareness without an extra round-trip, every successful /v1/chat/completions from a trial key also includes X-InferAll-Trial-Usage: N/CAP and X-InferAll-Trial-Status: active|halfway|near_cap response headers, where CAP is the allowance the gateway is currently enforcing. Read it from the header rather than hardcoding a number, since it is a server-side setting and has changed.

Providers

OpenAIGPT-4.1, o3, o4-mini, GPT-4o, gpt-image-1
AnthropicClaude Opus/Sonnet/Haiku 4.x
Google Gemini2.5 Flash/Pro, Veo, Imagen
NVIDIA NIM10+ free models (Llama, GLM 5.3, Nemotron)
ReplicateFlux, Stable Diffusion
RunwayGen-4.5, video generation
ElevenLabseleven_multilingual_v2 TTS

Live model list

Browse every model and its price on the model catalog, which marks the free ones a no-card trial key can call. The raw JSON is at api.inferall.ai/ai/v1/models. The OpenAI-style list at /v1/models gives each model a pricing object with the prompt and completion price you are billed per token, in USD, so a client can show what a call costs.

More resources

Integrations — Claude Code, Cline, Cursor, LangChain, LlamaIndex, and more

Pricing — Free trial, Pro, Team, and Enterprise plans

Rate limits — Per-operation limits and spending caps