Band I · Compute
When it fails, it fails like “too slow, too expensive”
Band I is layers Inference & Serving and Routing & Gateways. 22 tools sit here, and the failure mode is recognisable before you have read any of them.
Start from the symptom instead
Each is an ordered checklist through the stack, cheapest check first.
Who ends up owning it
A band is a property of the stack; a role is a property of a person. They correlate but are not the same question — which is why the count is derived rather than asserted.
The layers
01Inference & Serving
12 toolsTurns weights into throughput.. Runtimes that turn weights into tokens per second.
- vLLMTraffic is steady and low-concurrency, so batches never fill.
- SGLangYou want the fewest moving parts and vLLM already clears your bar.
- llama.cppYou need maximum concurrent throughput on datacentre GPUs.
- OllamaYou need production throughput or a stable server API.
- TensorRT-LLMHardware portability matters, or you have no GPUs to tune against.
- Text Generation InferenceThe model fits on one GPU, where this is overkill.
- LM StudioYou need headless, multi-tenant deployment.
- Triton Inference ServerYou serve one model family and want peak LLM throughput.
- llamafileYou need batching, multi-GPU serving or a production HTTP server.
- KoboldCppYou need headless serving at scale.
- PowerInferYou have GPUs to spare — it trades peak throughput for reach.
- MarlinYou need a scheduler and a server as well as a kernel.
02Routing & Gateways
10 toolsDecides which model answers, and what it costs.. One endpoint across many providers, with policy in between.
- LiteLLMYou want routing policy to live in a SaaS console.
- PortkeyPrompts cannot leave your infrastructure.
- Cloudflare AI GatewayRequests must not transit a third party's network.
- OpenRouterYou need self-hosting or per-provider SLAs.
- MartianYou want the routing rule to be plain config you can read.
- Envoy AI GatewayYou want a gateway configurable without a data plane.
- BifrostYou need the breadth of a full routing stack.
- RouteLLMYou cannot evaluate the quality you would be trading away.
- Not DiamondSelf-hosting or auditability is a requirement.
- TrueFoundryYou want to adopt the pieces independently.
Indexed elsewhere, but serving this band
The problem is in band I and the tool you need is filed under a different layer. These are the crossings, each with the reason it belongs here too — a bare cross-reference reads as a mistake, so the reason is the whole payload.
Bands are a shortcut, not a taxonomy — band I covers 2 of nine layers, so the band tells you where to look without telling you how much of the stack is involved. The method says where else this index is wrong.