> ## Documentation Index
> Fetch the complete documentation index at: https://runinfra.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Models and inference

> FAQ about supported models, quantization, context windows, streaming, tool calling, and fine-tuning.

<AccordionGroup>
  <Accordion title="Which model families are supported?">
    Llama 3.x/4, Qwen 2.5, Mistral and Mixtral, DeepSeek V2/V3/R1, Gemma 2, Phi-3/4, Cohere Command-R/R+, Whisper (all sizes), XTTS v2, Bark. Plus any compatible model on Hugging Face. See [Models](/features/models).
  </Accordion>

  <Accordion title="Can I use a custom fine-tuned model?">
    Yes. Upload SafeTensors, PyTorch, GGUF, or ONNX up to 50 GB at [runinfra.ai/models](https://runinfra.ai/models). The optimizer runs on custom models just like catalog models.
  </Accordion>

  <Accordion title="What context window does each model have?">
    Depends on the model. Llama 3.1 is 128K, Qwen 2.5 is 128K, Mistral Small 3 is 32K, DeepSeek V3 is 128K. The `/v1/models` endpoint returns each model's context window. Exceeding the window returns 400 `context_length_exceeded`.
  </Accordion>

  <Accordion title="Which quantization should I pick?">
    Let the optimizer rank compatible methods for your model, runtime, GPU, and quality target. 4-bit LLM variants reduce VRAM and FP8 can preserve more fidelity on compatible GPUs. See [Optimization](/features/optimization).
  </Accordion>

  <Accordion title="Does streaming work?">
    Yes for chat and Responses requests when the selected deployment supports streaming. Set `stream: true` and iterate chunks. See the [streaming cookbook](/cookbook/streaming).
  </Accordion>

  <Accordion title="Does tool/function calling work?">
    Yes. Use the OpenAI-shaped `tools` array and `tool_choice` parameter on compatible chat deployments. See the [tool calling cookbook](/cookbook/tool-calling).
  </Accordion>

  <Accordion title="Does structured output work?">
    Yes. `response_format` with `json_object` or `json_schema` (strict mode). Compatible with Pydantic and Zod helpers in the OpenAI SDK. See the [structured output cookbook](/cookbook/structured-output).
  </Accordion>

  <Accordion title="Does vision work?">
    Yes on VLM-backed pipelines (Llama 3.2 Vision, Qwen 2.5 VL, etc.). Send `image_url` or base64 content blocks. Up to 30 images per request.
  </Accordion>

  <Accordion title="Does RunInfra do fine-tuning?">
    Supervised fine-tuning and LoRA are on the roadmap. Today, fine-tune externally (HF Transformers, Axolotl, Unsloth) and upload the result as a custom model.
  </Accordion>

  <Accordion title="What's the maximum `max_tokens` I can request?">
    Up to the model's context window minus your prompt length. Practical limits: 8K for most chat flows, 16-32K for long summaries. If you need more, split into [streaming](/cookbook/streaming) requests to avoid 504 timeouts.
  </Accordion>

  <Accordion title="Is there batch inference?">
    Managed batch inference is not a public API today. The recommended pattern is async parallel requests with `AsyncOpenAI` and your own queue.
  </Accordion>

  <Accordion title="Can I compare models side-by-side?">
    Yes, two ways. Ask the agent to compare 2 to 5 models and RunInfra benchmarks them sequentially on the same GPU with the same serving configuration, then shows a measured side-by-side table with a winner; results appear as a comparison card in chat and in the Compare Results panel in the session workspace, and survive reload. See [Compare models, engines, and techniques](/features/optimization#compare-models-engines-and-techniques). You can also compare two variants manually in the playground under the Deploy tab by sending the same prompt to both.
  </Accordion>

  <Accordion title="How do I know which model my pipeline is serving?">
    `GET /v1/models` returns the callable model IDs visible to your key. The response of any chat completion also includes the `model` value that served the request.
  </Accordion>

  <Accordion title="Can I pin a specific model version?">
    Yes. Pass the full Hugging Face ID (`meta-llama/Llama-3.1-8B-Instruct`) in your request, or use a deployed alias returned by `GET /v1/models`. Pipelines support explicit aliases that map names to specific commits.
  </Accordion>
</AccordionGroup>

## Not here?

<Columns cols={2}>
  <Card title="Models catalog" icon="cpu" href="/features/models">
    Full supported list with parameters.
  </Card>

  <Card title="Optimization" icon="zap" href="/features/optimization">
    How quantization and Forge work.
  </Card>

  <Card title="API reference" icon="square-terminal" href="/api-reference/introduction">
    Endpoint parameter details.
  </Card>

  <Card title="Which model?" icon="compass" href="/introduction/model-picker">
    Decision table for picking a starting point.
  </Card>
</Columns>