Serving
Mixture-of-experts serving routes each token through a selected subset of available experts. Total parameter storage can greatly exceed the parameters active for one token.
Memory must hold or fetch the expert pool even though computation touches only selected experts. Routing balance and cross-device token movement can determine realized utilization.
Operators place experts across devices, set routing capacity, and observe token distribution under real prompts. They measure load skew, communication, cache use, and latency by expert path.
Mixture-of-experts serving routes each token through a selected subset of available experts. Total parameter storage can greatly exceed the parameters active for one token. Operators place experts across devices, set routing capacity, and observe token distribution under real prompts. They measure load skew, communication, cache use, and latency by expert path.
Memory must hold or fetch the expert pool even though computation touches only selected experts. Routing balance and cross-device token movement can determine realized utilization.
RunInfraby RightNow© 2026 RunInfra. All rights reserved.
