Skip to content

RunInfraby RightNow

Model APIs
Pricing
Research
Contact

Dashboard Sign in Get started

Home/Glossary/Agent inference

Glossary category

Agent inference glossary

Definitions for inference serving terms in the agent inference category.

Terms in this category

ANTHROPIC_BASE_URLANTHROPIC_BASE_URL is the environment variable that points Claude Code at a custom API endpoint.
Anthropic-compatible APIAn Anthropic-compatible API accepts requests in the Anthropic Messages format so clients can use another model provider.
OpenAI Responses APIThe OpenAI Responses API is an interface for generating model output from inputs and representing text, reasoning, and tool calls as response items.
Session affinitySession affinity is a routing policy that keeps requests from the same conversation together to improve reuse of cached prompt work.
Cached input tokensCached input tokens are prompt tokens whose stored computation is reused from a matching prefix instead of being recomputed.

Related glossary categories

Batching glossary
Memory glossary
Quantization glossary
Latency glossary
Parallelism glossary
Serving glossary
Hardware glossary

Call Nemotron 3.5 Lightning 30B with Model APIs

Use a workspace API key and pay for input, cached input, and output tokens.

View Model APIs

TLS in transit, AES-256 at rest

Workspace isolation

No training on your data

SOC 2 Type II

RunInfraby RightNow

© 2026 RunInfra. All rights reserved.

Coding plans Model APIs Pricing Startups Benchmarks Docs Research News Brand assets Contact

Backed by

Y Combinator

Ask AI about RunInfra

Part of RightNow

Security DPA AUP Cookies Terms Privacy