Llama-3.1-8B-Instruct does not fit on 1 x NVIDIA RTX 4000 Ada 20 GB: 19.7 GB required, 18 GB available.

Fit is always evaluated when model sizing is available. GPU rent, paid idle capacity, API crossover, and measured optimization appear only when their required evidence exists. When the API wins, the page says so.

Model

meta-llama/Llama-3.1-8B-Instruct

GPU

NVIDIA RTX 4000 Ada 20 GB

Requests/day

50,000

GPU count

1

Quantization

FP16

Tokens/request

500 in, 500 out

Charts appear only when evidence existsUtilization defaults to 30%

Configure the workload

Change the inputs, then calculate a shareable result.

Hugging Face model

GPUGPU countRequests per dayQuantizationAverage input tokensAverage output tokensMaximum served contextMaximum combined prompt and generated tokens accepted by the server. This is capacity, not average request length.

Does not fit

This configuration needs more memory.

The model and KV cache need more VRAM than the selected GPU count provides. Cost is intentionally hidden for a configuration that cannot run.

Model

meta-llama/Llama-3.1-8B-Instruct

Hardware

1 x NVIDIA RTX 4000 Ada 20 GB, FP16

Memory fit

Weights and KV cache stay separate so the fit is reproducible.

16.1 GB
Model weights: 1.1 GB
KV cache: 19.7 GB
Total required: 18 GB
Available

8,192 tokens
Served context used: 2.6 GB
Runtime overhead: Exact transformer shape
KV cache method

Memory-fit rental floors

Published rent floors for memory-fitting configurations.

Each row uses an engine GPU fit when present; the remaining catalog GPUs apply this result's required memory to published VRAM. GPU count rises only enough to clear memory.

Engine memory-fit result

Memory fit: 1x NVIDIA A10 24 GB at 90% vLLM GPU memory utilization.

On-demand rates are preferred for each GPU; when none is captured, the lowest dated captured tier is used. The rentable GPU catalog is ranked by one-deployment monthly rent over the contract's 730.5-hour month, rounded to cents. Equal displayed prices share a rank. These are arithmetic floors, not provider offers. Throughput, replica demand, traffic capacity, topology support, and current availability are not established.

Floor rank 1
1 x NVIDIA RTX A5000 24 GB
Memory fits
19.7 GB required, 21.6 GB available inside the engine memory budget.
Current result memory total, Ampere, 768 GB/s memory bandwidth
One-deployment monthly rent floor
$116.88
RunPod, Community Cloud, on demand
Captured 2026-08-08
Arithmetic floor for 1 GPU over the contract's 730.5-hour month. This is not a provider offer.
Open rental source
Floor rank 2
2 x NVIDIA RTX A4000 16 GB
Memory fits
19.7 GB required, 28.8 GB available inside the engine memory budget.
Current result memory total, Ampere, 448 GB/s memory bandwidth
One-deployment monthly rent floor
$219.15
Hyperstack, on demand
Captured 2026-08-08
Arithmetic floor for 2 GPUs over the contract's 730.5-hour month. This is not a provider offer.
Open rental source
Floor rank 3
1 x NVIDIA RTX A6000 48 GB
Memory fits
19.7 GB required, 43.2 GB available inside the engine memory budget.
Current result memory total, Ampere, 768 GB/s memory bandwidth
One-deployment monthly rent floor
$241.07
RunPod, Community Cloud, on demand
Captured 2026-08-08
Arithmetic floor for 1 GPU over the contract's 730.5-hour month. This is not a provider offer.
Open rental source

What to change

Computed actions and cost signals appear only when the result supports them.

RecommendationComputed memory fit

Consider FP8 or INT8 if the checkpoint and hardware support it

FP16 weights use 16.1 GB and bring total memory to 19.7 GB; FP8 or INT8 would lower weights to 8 GB and total memory to 9.1 GB, which fits inside 18 GB.

Show 2 more computed options

RecommendationComputed memory fit

Reduce context length or serving concurrency

Weights use 16.1 GB and fit inside 18 GB, but the 1.1 GB KV cache raises total memory to 19.7 GB; reduce --max-model-len or peak concurrency before adding hardware.

RecommendationComputed memory fit

Raise tensor parallel size to 2

The current vLLM command uses tensor parallel size 1 and provides 18 GB, while 19.7 GB requires 2 same-GPU workers for 36 GB; raise --tensor-parallel-size to 2 if the model topology supports it.

Decode bandwidth ceiling

A physics bound, not an expected throughput result.

22 output tok/sCeiling

Bandwidth decode CEILING, not expected throughput. Ideal tensor-parallel scaling is assumed. Real decode throughput is lower.

Decode ceiling derived from GPU memory bandwidth and model weight bytes, as of 2026-08-08

Reproduce with vLLM

The exact command attached to this result.

vllm serve 'meta-llama/Llama-3.1-8B-Instruct' --dtype float16 --tensor-parallel-size 1 --max-model-len 8192 --gpu-memory-utilization 0.9

Sources and caveats

Every published number keeps its source and date.

Provenance

Hugging Face model API revision sha
As of 2026-08-08
Hugging Face model API safetensors.total
As of 2026-08-08
Hugging Face config.json
As of 2026-08-08
NVIDIA RTX 4000 Ada Generation datasheet
As of 2026-08-08
Meta Llama 3.1 8B Hugging Face config
As of 2026-08-08
RunInfra Engine feasibility runtime-overhead policy
As of 2026-08-09
Decode ceiling derived from GPU memory bandwidth and model weight bytes
As of 2026-08-08
vLLM serve CLI documentation
As of 2026-08-08
NVIDIA L4 product specification
As of 2026-08-08
NVIDIA A10 datasheet
As of 2026-08-08
NVIDIA L40S product specification
As of 2026-08-08
NVIDIA A100 product specification
As of 2026-08-08

Caveats

Served context is 8192 tokens. It is a serving-capacity decision and is not inferred from average input or output tokens.
KV cache uses the exact transformer-shape formula with 32 layers, 8 KV heads, and head dimension 128.
Runtime overhead adds 2.57 GB to totalRequiredGb. This approximate 15% policy allowance covers CUDA context, activations, and vLLM overhead. Deployment-time profiling, including CUDA graph capture and allocator behavior, can differ.
Bandwidth decode CEILING, not expected throughput. Ideal tensor-parallel scaling is assumed. Real decode throughput is lower.
No dense compute ceiling is cited for fp16 on NVIDIA RTX 4000 Ada 20 GB. The available physics bound is decode-only, so the entry point returns no cost instead of copying decode throughput into the input field.
The selected configuration does not fit, so no cost is returned.
The Hugging Face repository is gated. Running the generated command requires authorized access to the model weights.

Does not fit

This configuration needs more memory.

The model and KV cache need more VRAM than the selected GPU count provides. Cost is intentionally hidden for a configuration that cannot run.

Model

meta-llama/Llama-3.1-8B-Instruct

Hardware

1 x NVIDIA RTX 4000 Ada 20 GB, FP16

Memory fit

Weights and KV cache stay separate so the fit is reproducible.

16.1 GB
Model weights: 1.1 GB
KV cache: 19.7 GB
Total required: 18 GB
Available

8,192 tokens
Served context used: 2.6 GB
Runtime overhead: Exact transformer shape
KV cache method

Memory-fit rental floors

Published rent floors for memory-fitting configurations.

Each row uses an engine GPU fit when present; the remaining catalog GPUs apply this result's required memory to published VRAM. GPU count rises only enough to clear memory.

Engine memory-fit result

Memory fit: 1x NVIDIA A10 24 GB at 90% vLLM GPU memory utilization.

On-demand rates are preferred for each GPU; when none is captured, the lowest dated captured tier is used. The rentable GPU catalog is ranked by one-deployment monthly rent over the contract's 730.5-hour month, rounded to cents. Equal displayed prices share a rank. These are arithmetic floors, not provider offers. Throughput, replica demand, traffic capacity, topology support, and current availability are not established.

Floor rank 1
1 x NVIDIA RTX A5000 24 GB
Memory fits
19.7 GB required, 21.6 GB available inside the engine memory budget.
Current result memory total, Ampere, 768 GB/s memory bandwidth
One-deployment monthly rent floor
$116.88
RunPod, Community Cloud, on demand
Captured 2026-08-08
Arithmetic floor for 1 GPU over the contract's 730.5-hour month. This is not a provider offer.
Open rental source
Floor rank 2
2 x NVIDIA RTX A4000 16 GB
Memory fits
19.7 GB required, 28.8 GB available inside the engine memory budget.
Current result memory total, Ampere, 448 GB/s memory bandwidth
One-deployment monthly rent floor
$219.15
Hyperstack, on demand
Captured 2026-08-08
Arithmetic floor for 2 GPUs over the contract's 730.5-hour month. This is not a provider offer.
Open rental source
Floor rank 3
1 x NVIDIA RTX A6000 48 GB
Memory fits
19.7 GB required, 43.2 GB available inside the engine memory budget.
Current result memory total, Ampere, 768 GB/s memory bandwidth
One-deployment monthly rent floor
$241.07
RunPod, Community Cloud, on demand
Captured 2026-08-08
Arithmetic floor for 1 GPU over the contract's 730.5-hour month. This is not a provider offer.
Open rental source

What to change

Computed actions and cost signals appear only when the result supports them.

RecommendationComputed memory fit

Consider FP8 or INT8 if the checkpoint and hardware support it

FP16 weights use 16.1 GB and bring total memory to 19.7 GB; FP8 or INT8 would lower weights to 8 GB and total memory to 9.1 GB, which fits inside 18 GB.

Show 2 more computed options

RecommendationComputed memory fit

Reduce context length or serving concurrency

Weights use 16.1 GB and fit inside 18 GB, but the 1.1 GB KV cache raises total memory to 19.7 GB; reduce --max-model-len or peak concurrency before adding hardware.

RecommendationComputed memory fit

Raise tensor parallel size to 2

The current vLLM command uses tensor parallel size 1 and provides 18 GB, while 19.7 GB requires 2 same-GPU workers for 36 GB; raise --tensor-parallel-size to 2 if the model topology supports it.

Decode bandwidth ceiling

A physics bound, not an expected throughput result.

22 output tok/sCeiling

Bandwidth decode CEILING, not expected throughput. Ideal tensor-parallel scaling is assumed. Real decode throughput is lower.

Decode ceiling derived from GPU memory bandwidth and model weight bytes, as of 2026-08-08

Reproduce with vLLM

The exact command attached to this result.

vllm serve 'meta-llama/Llama-3.1-8B-Instruct' --dtype float16 --tensor-parallel-size 1 --max-model-len 8192 --gpu-memory-utilization 0.9

Sources and caveats

Every published number keeps its source and date.

Provenance

Hugging Face model API revision sha
As of 2026-08-08
Hugging Face model API safetensors.total
As of 2026-08-08
Hugging Face config.json
As of 2026-08-08
NVIDIA RTX 4000 Ada Generation datasheet
As of 2026-08-08
Meta Llama 3.1 8B Hugging Face config
As of 2026-08-08
RunInfra Engine feasibility runtime-overhead policy
As of 2026-08-09
Decode ceiling derived from GPU memory bandwidth and model weight bytes
As of 2026-08-08
vLLM serve CLI documentation
As of 2026-08-08
NVIDIA L4 product specification
As of 2026-08-08
NVIDIA A10 datasheet
As of 2026-08-08
NVIDIA L40S product specification
As of 2026-08-08
NVIDIA A100 product specification
As of 2026-08-08

Caveats

Served context is 8192 tokens. It is a serving-capacity decision and is not inferred from average input or output tokens.
KV cache uses the exact transformer-shape formula with 32 layers, 8 KV heads, and head dimension 128.
Runtime overhead adds 2.57 GB to totalRequiredGb. This approximate 15% policy allowance covers CUDA context, activations, and vLLM overhead. Deployment-time profiling, including CUDA graph capture and allocator behavior, can differ.
Bandwidth decode CEILING, not expected throughput. Ideal tensor-parallel scaling is assumed. Real decode throughput is lower.
No dense compute ceiling is cited for fp16 on NVIDIA RTX 4000 Ada 20 GB. The available physics bound is decode-only, so the entry point returns no cost instead of copying decode throughput into the input field.
The selected configuration does not fit, so no cost is returned.
The Hugging Face repository is gated. Running the generated command requires authorized access to the model weights.

Does not fit

This configuration needs more memory.

The model and KV cache need more VRAM than the selected GPU count provides. Cost is intentionally hidden for a configuration that cannot run.

Model

meta-llama/Llama-3.1-8B-Instruct

Hardware

1 x NVIDIA RTX 4000 Ada 20 GB, FP16

Memory fit

Weights and KV cache stay separate so the fit is reproducible.

16.1 GB
Model weights: 1.1 GB
KV cache: 19.7 GB
Total required: 18 GB
Available

8,192 tokens
Served context used: 2.6 GB
Runtime overhead: Exact transformer shape
KV cache method

Memory-fit rental floors

Published rent floors for memory-fitting configurations.

Each row uses an engine GPU fit when present; the remaining catalog GPUs apply this result's required memory to published VRAM. GPU count rises only enough to clear memory.

Engine memory-fit result

Memory fit: 1x NVIDIA A10 24 GB at 90% vLLM GPU memory utilization.

On-demand rates are preferred for each GPU; when none is captured, the lowest dated captured tier is used. The rentable GPU catalog is ranked by one-deployment monthly rent over the contract's 730.5-hour month, rounded to cents. Equal displayed prices share a rank. These are arithmetic floors, not provider offers. Throughput, replica demand, traffic capacity, topology support, and current availability are not established.

Floor rank 1
1 x NVIDIA RTX A5000 24 GB
Memory fits
19.7 GB required, 21.6 GB available inside the engine memory budget.
Current result memory total, Ampere, 768 GB/s memory bandwidth
One-deployment monthly rent floor
$116.88
RunPod, Community Cloud, on demand
Captured 2026-08-08
Arithmetic floor for 1 GPU over the contract's 730.5-hour month. This is not a provider offer.
Open rental source
Floor rank 2
2 x NVIDIA RTX A4000 16 GB
Memory fits
19.7 GB required, 28.8 GB available inside the engine memory budget.
Current result memory total, Ampere, 448 GB/s memory bandwidth
One-deployment monthly rent floor
$219.15
Hyperstack, on demand
Captured 2026-08-08
Arithmetic floor for 2 GPUs over the contract's 730.5-hour month. This is not a provider offer.
Open rental source
Floor rank 3
1 x NVIDIA RTX A6000 48 GB
Memory fits
19.7 GB required, 43.2 GB available inside the engine memory budget.
Current result memory total, Ampere, 768 GB/s memory bandwidth
One-deployment monthly rent floor
$241.07
RunPod, Community Cloud, on demand
Captured 2026-08-08
Arithmetic floor for 1 GPU over the contract's 730.5-hour month. This is not a provider offer.
Open rental source

What to change

Computed actions and cost signals appear only when the result supports them.

RecommendationComputed memory fit

Consider FP8 or INT8 if the checkpoint and hardware support it

FP16 weights use 16.1 GB and bring total memory to 19.7 GB; FP8 or INT8 would lower weights to 8 GB and total memory to 9.1 GB, which fits inside 18 GB.

Show 2 more computed options

RecommendationComputed memory fit

Reduce context length or serving concurrency

Weights use 16.1 GB and fit inside 18 GB, but the 1.1 GB KV cache raises total memory to 19.7 GB; reduce --max-model-len or peak concurrency before adding hardware.

RecommendationComputed memory fit

Raise tensor parallel size to 2

The current vLLM command uses tensor parallel size 1 and provides 18 GB, while 19.7 GB requires 2 same-GPU workers for 36 GB; raise --tensor-parallel-size to 2 if the model topology supports it.

Decode bandwidth ceiling

A physics bound, not an expected throughput result.

22 output tok/sCeiling

Bandwidth decode CEILING, not expected throughput. Ideal tensor-parallel scaling is assumed. Real decode throughput is lower.

Decode ceiling derived from GPU memory bandwidth and model weight bytes, as of 2026-08-08

Reproduce with vLLM

The exact command attached to this result.

vllm serve 'meta-llama/Llama-3.1-8B-Instruct' --dtype float16 --tensor-parallel-size 1 --max-model-len 8192 --gpu-memory-utilization 0.9

Sources and caveats

Every published number keeps its source and date.

Provenance

Hugging Face model API revision sha
As of 2026-08-08
Hugging Face model API safetensors.total
As of 2026-08-08
Hugging Face config.json
As of 2026-08-08
NVIDIA RTX 4000 Ada Generation datasheet
As of 2026-08-08
Meta Llama 3.1 8B Hugging Face config
As of 2026-08-08
RunInfra Engine feasibility runtime-overhead policy
As of 2026-08-09
Decode ceiling derived from GPU memory bandwidth and model weight bytes
As of 2026-08-08
vLLM serve CLI documentation
As of 2026-08-08
NVIDIA L4 product specification
As of 2026-08-08
NVIDIA A10 datasheet
As of 2026-08-08
NVIDIA L40S product specification
As of 2026-08-08
NVIDIA A100 product specification
As of 2026-08-08

Caveats

Served context is 8192 tokens. It is a serving-capacity decision and is not inferred from average input or output tokens.
KV cache uses the exact transformer-shape formula with 32 layers, 8 KV heads, and head dimension 128.
Runtime overhead adds 2.57 GB to totalRequiredGb. This approximate 15% policy allowance covers CUDA context, activations, and vLLM overhead. Deployment-time profiling, including CUDA graph capture and allocator behavior, can differ.
Bandwidth decode CEILING, not expected throughput. Ideal tensor-parallel scaling is assumed. Real decode throughput is lower.
No dense compute ceiling is cited for fp16 on NVIDIA RTX 4000 Ada 20 GB. The available physics bound is decode-only, so the entry point returns no cost instead of copying decode throughput into the input field.
The selected configuration does not fit, so no cost is returned.
The Hugging Face repository is gated. Running the generated command requires authorized access to the model weights.

Does not fit

This configuration needs more memory.

The model and KV cache need more VRAM than the selected GPU count provides. Cost is intentionally hidden for a configuration that cannot run.

Model

meta-llama/Llama-3.1-8B-Instruct

Hardware

1 x NVIDIA RTX 4000 Ada 20 GB, FP16

Memory fit

Weights and KV cache stay separate so the fit is reproducible.

16.1 GB
Model weights: 1.1 GB
KV cache: 19.7 GB
Total required: 18 GB
Available

8,192 tokens
Served context used: 2.6 GB
Runtime overhead: Exact transformer shape
KV cache method

Memory-fit rental floors

Published rent floors for memory-fitting configurations.

Each row uses an engine GPU fit when present; the remaining catalog GPUs apply this result's required memory to published VRAM. GPU count rises only enough to clear memory.

Engine memory-fit result

Memory fit: 1x NVIDIA A10 24 GB at 90% vLLM GPU memory utilization.

On-demand rates are preferred for each GPU; when none is captured, the lowest dated captured tier is used. The rentable GPU catalog is ranked by one-deployment monthly rent over the contract's 730.5-hour month, rounded to cents. Equal displayed prices share a rank. These are arithmetic floors, not provider offers. Throughput, replica demand, traffic capacity, topology support, and current availability are not established.

Floor rank 1
1 x NVIDIA RTX A5000 24 GB
Memory fits
19.7 GB required, 21.6 GB available inside the engine memory budget.
Current result memory total, Ampere, 768 GB/s memory bandwidth
One-deployment monthly rent floor
$116.88
RunPod, Community Cloud, on demand
Captured 2026-08-08
Arithmetic floor for 1 GPU over the contract's 730.5-hour month. This is not a provider offer.
Open rental source
Floor rank 2
2 x NVIDIA RTX A4000 16 GB
Memory fits
19.7 GB required, 28.8 GB available inside the engine memory budget.
Current result memory total, Ampere, 448 GB/s memory bandwidth
One-deployment monthly rent floor
$219.15
Hyperstack, on demand
Captured 2026-08-08
Arithmetic floor for 2 GPUs over the contract's 730.5-hour month. This is not a provider offer.
Open rental source
Floor rank 3
1 x NVIDIA RTX A6000 48 GB
Memory fits
19.7 GB required, 43.2 GB available inside the engine memory budget.
Current result memory total, Ampere, 768 GB/s memory bandwidth
One-deployment monthly rent floor
$241.07
RunPod, Community Cloud, on demand
Captured 2026-08-08
Arithmetic floor for 1 GPU over the contract's 730.5-hour month. This is not a provider offer.
Open rental source

What to change

Computed actions and cost signals appear only when the result supports them.

RecommendationComputed memory fit

Consider FP8 or INT8 if the checkpoint and hardware support it

FP16 weights use 16.1 GB and bring total memory to 19.7 GB; FP8 or INT8 would lower weights to 8 GB and total memory to 9.1 GB, which fits inside 18 GB.

Show 2 more computed options

RecommendationComputed memory fit

Reduce context length or serving concurrency

Weights use 16.1 GB and fit inside 18 GB, but the 1.1 GB KV cache raises total memory to 19.7 GB; reduce --max-model-len or peak concurrency before adding hardware.

RecommendationComputed memory fit

Raise tensor parallel size to 2

The current vLLM command uses tensor parallel size 1 and provides 18 GB, while 19.7 GB requires 2 same-GPU workers for 36 GB; raise --tensor-parallel-size to 2 if the model topology supports it.

Decode bandwidth ceiling

A physics bound, not an expected throughput result.

22 output tok/sCeiling

Bandwidth decode CEILING, not expected throughput. Ideal tensor-parallel scaling is assumed. Real decode throughput is lower.

Decode ceiling derived from GPU memory bandwidth and model weight bytes, as of 2026-08-08

Reproduce with vLLM

The exact command attached to this result.

vllm serve 'meta-llama/Llama-3.1-8B-Instruct' --dtype float16 --tensor-parallel-size 1 --max-model-len 8192 --gpu-memory-utilization 0.9

Sources and caveats

Every published number keeps its source and date.

Provenance

Hugging Face model API revision sha
As of 2026-08-08
Hugging Face model API safetensors.total
As of 2026-08-08
Hugging Face config.json
As of 2026-08-08
NVIDIA RTX 4000 Ada Generation datasheet
As of 2026-08-08
Meta Llama 3.1 8B Hugging Face config
As of 2026-08-08
RunInfra Engine feasibility runtime-overhead policy
As of 2026-08-09
Decode ceiling derived from GPU memory bandwidth and model weight bytes
As of 2026-08-08
vLLM serve CLI documentation
As of 2026-08-08
NVIDIA L4 product specification
As of 2026-08-08
NVIDIA A10 datasheet
As of 2026-08-08
NVIDIA L40S product specification
As of 2026-08-08
NVIDIA A100 product specification
As of 2026-08-08

Caveats

Served context is 8192 tokens. It is a serving-capacity decision and is not inferred from average input or output tokens.
KV cache uses the exact transformer-shape formula with 32 layers, 8 KV heads, and head dimension 128.
Runtime overhead adds 2.57 GB to totalRequiredGb. This approximate 15% policy allowance covers CUDA context, activations, and vLLM overhead. Deployment-time profiling, including CUDA graph capture and allocator behavior, can differ.
Bandwidth decode CEILING, not expected throughput. Ideal tensor-parallel scaling is assumed. Real decode throughput is lower.
No dense compute ceiling is cited for fp16 on NVIDIA RTX 4000 Ada 20 GB. The available physics bound is decode-only, so the entry point returns no cost instead of copying decode throughput into the input field.
The selected configuration does not fit, so no cost is returned.
The Hugging Face repository is gated. Running the generated command requires authorized access to the model weights.

Share this calculation

Embed code appears only when the calculator can publish a sourced cost for a fitting configuration.

Badge unavailable for this configuration

This configuration does not fit the selected hardware, so the calculator does not publish a cost or embed code.

Make this configuration fit, describe what you need

End-to-end encryption

Isolated GPU infrastructure

No training on your data

SOC 2 Type II

Configure the workload

This configuration needs more memory.

Memory fit

Published rent floors for memory-fitting configurations.

1 x NVIDIA RTX A5000 24 GB

2 x NVIDIA RTX A4000 16 GB

1 x NVIDIA RTX A6000 48 GB

What to change

Consider FP8 or INT8 if the checkpoint and hardware support it

Reduce context length or serving concurrency

Raise tensor parallel size to 2

Decode bandwidth ceiling

Reproduce with vLLM

Sources and caveats

Provenance

Caveats

This configuration needs more memory.

Memory fit

Published rent floors for memory-fitting configurations.

1 x NVIDIA RTX A5000 24 GB

2 x NVIDIA RTX A4000 16 GB

1 x NVIDIA RTX A6000 48 GB

What to change

Consider FP8 or INT8 if the checkpoint and hardware support it

Reduce context length or serving concurrency

Raise tensor parallel size to 2

Decode bandwidth ceiling

Reproduce with vLLM

Sources and caveats

Provenance

Caveats

This configuration needs more memory.

Memory fit

Published rent floors for memory-fitting configurations.

1 x NVIDIA RTX A5000 24 GB

2 x NVIDIA RTX A4000 16 GB

1 x NVIDIA RTX A6000 48 GB

What to change

Consider FP8 or INT8 if the checkpoint and hardware support it

Reduce context length or serving concurrency

Raise tensor parallel size to 2

Decode bandwidth ceiling

Reproduce with vLLM

Sources and caveats

Provenance

Caveats

This configuration needs more memory.

Memory fit

Published rent floors for memory-fitting configurations.

1 x NVIDIA RTX A5000 24 GB

2 x NVIDIA RTX A4000 16 GB

1 x NVIDIA RTX A6000 48 GB

What to change

Consider FP8 or INT8 if the checkpoint and hardware support it

Reduce context length or serving concurrency

Raise tensor parallel size to 2

Decode bandwidth ceiling

Reproduce with vLLM

Sources and caveats

Provenance

Caveats

Share this calculation

Make this configuration fit, describe what you need

Configure the workload

This configuration needs more memory.

Memory fit

Published rent floors for memory-fitting configurations.

1 x NVIDIA RTX A5000 24 GB

2 x NVIDIA RTX A4000 16 GB

1 x NVIDIA RTX A6000 48 GB

What to change

Consider FP8 or INT8 if the checkpoint and hardware support it

Reduce context length or serving concurrency

Raise tensor parallel size to 2

Decode bandwidth ceiling

Reproduce with vLLM

Sources and caveats

Provenance

Caveats

This configuration needs more memory.