Does this support tool calling?

Yes. The chat completions API supports OpenAI-compatible tool calling and structured output across vLLM, SGLang, and TensorRT-LLM.

Can I run this on a smaller GPU?

Yes. Drop to a single L4 (24 GB) by switching to Llama 3.2 3B or Phi-3 Mini. The agent rebenchmarks and picks the GPU for your model.

RunInfraby RightNow

Dashboard Sign in Get started

Personal AI assistants on the models you own.

Llama, Hermes, Qwen, or supported open Hugging Face chat models. Tool use, policy, and streaming on a single GPU.

Deploy this pipeline Read the stack

Chat

Tool use

Code agent

Knowledge work

What you actually own

The optimization knobs, the codebase, the model choice. None of it locked away.

Pick the model your product needs.

Llama, Mistral, Qwen, Phi, Gemma, or your own fine-tune. The agent retunes the kernels around your choice.

Tool use and streaming on your stack.

OpenAI-compatible chat completions with tool calling and streaming. No proprietary client.

Own the runtime and the data.

Export the Dockerfile and serve script. Run on managed RunInfra or your own GPUs.

Three ways to ship a chat assistant

Most teams pick between speed and control. RunInfra keeps both in one workflow.

Deployment comparison for ai assistant across RunInfra, closed APIs, and DIY self-hosting.
What matters	RunInfraRecommendedFast path with model control and export.	Closed chat APIsPer-token, hosted.	DIY self-hostingFull control, heavy operations.
01Launch	Pick model, optimize, deploy Start quickly and keep the production path open.	Call provider endpoint Fast first demo, but the runtime stays rented.	Build serving stack first Infrastructure work comes before product learning.
02Model control	Bring the model ID Keep model choice and serving decisions visible.	Provider catalog You use what the provider exposes.	Your model Full control if your team maintains the runtime.
03Tuning	Measured latency and GPU cost Compare serving choices before deployment.	Opaque Latency and batching stay behind the API.	Manual profiling Your team owns tuning and regressions.
04Export	Managed now, export when needed Use the endpoint first and take the deploy package later.	Locked endpoint You keep calling the provider.	Already owned Export exists because you built everything yourself.
05Operations	Low until you choose to own it Operate managed, then export with the same measured plan.	Low, with lock-in Less infra work, less production control.	High You own infra, failures, upgrades, and serving changes.
06Security	SOC 2 Type 2 Audited controls across access, logging, and incident response.	Varies by vendor Compliance depends on the third party sitting in the request path.	You build it Your team owns the audit trail, logging, and access controls.

RunInfra

Recommended

Fast path with model control and export.

Launch

Pick model, optimize, deploy

Start quickly and keep the production path open.

Model control

Bring the model ID

Keep model choice and serving decisions visible.

Tuning

Measured latency and GPU cost

Compare serving choices before deployment.

Export

Managed now, export when needed

Use the endpoint first and take the deploy package later.

Operations

Low until you choose to own it

Operate managed, then export with the same measured plan.

Security

SOC 2 Type 2

Audited controls across access, logging, and incident response.

Code you own. Deploy anywhere.

The full recipe ships with you. Codebase, kernels, engine config, weights. Run it anywhere.

Live

Build a personal AI assistant. @Llama-3.1-8B for chat with tool calling. Serve on a single @L40S with streaming.

Agent

On it. I'll profile Llama on the L40S, configure vLLM with tool calling and streaming, then benchmark a realistic chat trace.

Profiled Llama-3.1-8B on L40S

85ms first-token p50

Selected vLLM serving engine

OpenAI-compatible chat with tool calling

Tuned AWQ INT4 quantization

+58% throughput at flat quality

Enabled prefix cache for tool prompts

0.4x KV cost on common turns

Ran chat harness, 500 turns plus tools

p95 380ms, 12k tok/s steady

Try Mistral 7B and rerun the harness...

Tool definitions3

runinfra-ai-assistant/

models/

evals/

Managed RunInfra

Our GPUs, per-million-tokens billing from L4 to B200.

Your infrastructure

AWS, GCP, RunPod, bare metal. Same Dockerfile, your cluster.

Local workstation

docker compose up. Full pipeline on a single GPU.

Supported HF chat models work

Supported chat-capable models on Hugging Face run through the compatible recipe. Search the live catalog above. The examples below are just a starting view.

HF

Llama 4 Maverick

Meta

400B MoEMoE

Llama 4 Scout

Meta

109B MoEMoE

Llama 3.3 70B

Meta

70BLLM

Llama 3.1 70B

Meta

70BLLM

Llama 3.1 8B

Meta

8BLLM

Llama 3.2 3B

Meta

3BEdge LLM

Llama 3.2 1B

Meta

1BEdge LLM

Hermes 3 (Llama 3.1 8B)

Nous Research

8BLLM

Qwen 2.5 72B

Alibaba

72BLLM

What RunInfra tunes

Every stage of the pipeline, retuned per model and GPU.

Kernel sweep

FlashAttention, FlashInfer, Marlin, custom fusion. Correctness + speedup gates on every rewrite.

Quantization

FP8, AWQ, GPTQ, HQQ, INT8 SmoothQuant, NVFP4. Quality scored against your accuracy floor.

KV cache

FP8, INT4, and TurboQuant 3-4 bit KV compression. 60 to 75% VRAM savings.

Speculative decoding

EAGLE3, MTP, n-gram lookup, draft model. 1.3 to 2x decode speedup, weights untouched.

Serving config

Continuous batching, chunked prefill, PagedAttention, prefix cache. Tuned across vLLM, SGLang, TensorRT-LLM.

Multi-cloud capacity

Pareto GPU selection across L4 to B200. Managed or exportable.

Try this pipeline

Edit the model, engine, or GPU inline. Send to retune the stack in the dashboard.

Common questions

Can't find what you're looking for? Get in touch

Can I bring my own fine-tuned model?

Yes, for supported chat-capable models. Paste the Hugging Face ID; unsupported or untuned shapes show baseline-only or gated status instead of a tuned run.

Deploy your first optimized model, measured before you ship

Describe the goal. RunInfra builds and optimizes the stack.

Start Building View Pricing

End-to-end encryption

Isolated GPU infrastructure

Zero data retention

SOC 2 Type II

RunInfraby RightNow

All systems operational

Backed by

Combinator

AICPA Type II

SOC 2

Ask AI about RunInfra

Part of RightNow

Personal AI assistants on the models you own.

Llama, Hermes, Qwen, or supported open Hugging Face chat models. Tool use, policy, and streaming on a single GPU.

Deploy this pipeline Read the stack

Chat

Tool use

Code agent

Knowledge work

What you actually own

The optimization knobs, the codebase, the model choice. None of it locked away.

Pick the model your product needs.

Llama, Mistral, Qwen, Phi, Gemma, or your own fine-tune. The agent retunes the kernels around your choice.

Tool use and streaming on your stack.

OpenAI-compatible chat completions with tool calling and streaming. No proprietary client.

Own the runtime and the data.

Export the Dockerfile and serve script. Run on managed RunInfra or your own GPUs.

Three ways to ship a chat assistant

Most teams pick between speed and control. RunInfra keeps both in one workflow.

Deployment comparison for ai assistant across RunInfra, closed APIs, and DIY self-hosting.
What matters	RunInfraRecommendedFast path with model control and export.	Closed chat APIsPer-token, hosted.	DIY self-hostingFull control, heavy operations.
01Launch	Pick model, optimize, deploy Start quickly and keep the production path open.	Call provider endpoint Fast first demo, but the runtime stays rented.	Build serving stack first Infrastructure work comes before product learning.
02Model control	Bring the model ID Keep model choice and serving decisions visible.	Provider catalog You use what the provider exposes.	Your model Full control if your team maintains the runtime.
03Tuning	Measured latency and GPU cost Compare serving choices before deployment.	Opaque Latency and batching stay behind the API.	Manual profiling Your team owns tuning and regressions.
04Export	Managed now, export when needed Use the endpoint first and take the deploy package later.	Locked endpoint You keep calling the provider.	Already owned Export exists because you built everything yourself.
05Operations	Low until you choose to own it Operate managed, then export with the same measured plan.	Low, with lock-in Less infra work, less production control.	High You own infra, failures, upgrades, and serving changes.
06Security	SOC 2 Type 2 Audited controls across access, logging, and incident response.	Varies by vendor Compliance depends on the third party sitting in the request path.	You build it Your team owns the audit trail, logging, and access controls.

RunInfra

Recommended

Fast path with model control and export.

Launch

Pick model, optimize, deploy

Start quickly and keep the production path open.

Model control

Bring the model ID

Keep model choice and serving decisions visible.

Tuning

Measured latency and GPU cost

Compare serving choices before deployment.

Export

Managed now, export when needed

Use the endpoint first and take the deploy package later.

Operations

Low until you choose to own it

Operate managed, then export with the same measured plan.

Security

SOC 2 Type 2

Audited controls across access, logging, and incident response.

Code you own. Deploy anywhere.

The full recipe ships with you. Codebase, kernels, engine config, weights. Run it anywhere.

Live

Build a personal AI assistant. @Llama-3.1-8B for chat with tool calling. Serve on a single @L40S with streaming.

Agent

On it. I'll profile Llama on the L40S, configure vLLM with tool calling and streaming, then benchmark a realistic chat trace.

Profiled Llama-3.1-8B on L40S

85ms first-token p50

Selected vLLM serving engine

OpenAI-compatible chat with tool calling

Tuned AWQ INT4 quantization

+58% throughput at flat quality

Enabled prefix cache for tool prompts

0.4x KV cost on common turns

Ran chat harness, 500 turns plus tools

p95 380ms, 12k tok/s steady

Try Mistral 7B and rerun the harness...

Tool definitions3

runinfra-ai-assistant/

models/

evals/

Managed RunInfra

Our GPUs, per-million-tokens billing from L4 to B200.

Your infrastructure

AWS, GCP, RunPod, bare metal. Same Dockerfile, your cluster.

Local workstation

docker compose up. Full pipeline on a single GPU.

Supported HF chat models work

Supported chat-capable models on Hugging Face run through the compatible recipe. Search the live catalog above. The examples below are just a starting view.

HF

Llama 4 Maverick

Meta

400B MoEMoE

Llama 4 Scout

Meta

109B MoEMoE

Llama 3.3 70B

Meta

70BLLM

Llama 3.1 70B

Meta

70BLLM

Llama 3.1 8B

Meta

8BLLM

Llama 3.2 3B

Meta

3BEdge LLM

Llama 3.2 1B

Meta

1BEdge LLM

Hermes 3 (Llama 3.1 8B)

Nous Research

8BLLM

Qwen 2.5 72B

Alibaba

72BLLM

What RunInfra tunes

Every stage of the pipeline, retuned per model and GPU.

Kernel sweep

FlashAttention, FlashInfer, Marlin, custom fusion. Correctness + speedup gates on every rewrite.

Quantization

FP8, AWQ, GPTQ, HQQ, INT8 SmoothQuant, NVFP4. Quality scored against your accuracy floor.

KV cache

FP8, INT4, and TurboQuant 3-4 bit KV compression. 60 to 75% VRAM savings.

Speculative decoding

EAGLE3, MTP, n-gram lookup, draft model. 1.3 to 2x decode speedup, weights untouched.

Serving config

Continuous batching, chunked prefill, PagedAttention, prefix cache. Tuned across vLLM, SGLang, TensorRT-LLM.

Multi-cloud capacity

Pareto GPU selection across L4 to B200. Managed or exportable.

Try this pipeline

Edit the model, engine, or GPU inline. Send to retune the stack in the dashboard.

Common questions

Can't find what you're looking for? Get in touch

Can I bring my own fine-tuned model?

Yes, for supported chat-capable models. Paste the Hugging Face ID; unsupported or untuned shapes show baseline-only or gated status instead of a tuned run.

Deploy your first optimized model, measured before you ship

Describe the goal. RunInfra builds and optimizes the stack.

Start Building View Pricing

End-to-end encryption

Isolated GPU infrastructure

Zero data retention

SOC 2 Type II

RunInfraby RightNow

All systems operational

Backed by

Combinator

AICPA Type II

SOC 2

Ask AI about RunInfra

Part of RightNow