Memory
Context length is the maximum number of token positions a serving configuration admits for a sequence. The model-supported limit and the server-configured limit can differ.
Longer sequences reserve more KV cache and can reduce the number of requests admitted together. Prompt and generation budgets must fit inside the same configured limit.
Admission logic checks requested tokens against model, server, and available-memory limits. Operators set the limit from validated model behavior and realistic cache headroom.
Context length is the maximum number of token positions a serving configuration admits for a sequence. The model-supported limit and the server-configured limit can differ. Admission logic checks requested tokens against model, server, and available-memory limits. Operators set the limit from validated model behavior and realistic cache headroom.
Longer sequences reserve more KV cache and can reduce the number of requests admitted together. Prompt and generation budgets must fit inside the same configured limit.
RunInfraby RightNow© 2026 RunInfra. All rights reserved.
