Intro
In the second step of the model iteration process, the goal is to decide whether a proposed model change can survive the infrastructure around it. A model may look promising in research validation, but still fail once it touches training resources, inference engines, API contracts, request routing, latency budgets, or cost constraints.
For example, multi-token prediction may look attractive because it can improve inference speed. But before it reaches QA, the infra team still needs to answer several practical questions: does the current serving framework support it, does the GPU setup benefit from it, does it change memory pressure, and does it affect the service contract exposed to the backend?
This post focuses on LLM products and LLM-powered services. Other ML systems may share the same validation mindset, but the concrete infra questions can be very different. If you have not read the series intro yet, it may be useful context before this post.
I will cover four topics:
- The distinction between dev infra and QA/prod infra
- The difference between model-based and service-based infra validation
- The main layers of latency testing
- The cost checks that should happen before QA
What Infra Validation Means
When discussing LLM infrastructure, we need to separate the infra used during development from the infra used in staging or production.
QA and production infra should be efficient, observable, and robust. Dev infra has a different job: it should expose whether a model or service change is compatible with the current system, and whether the system needs to change before the model can move forward.
In this stage, infra validation usually has two tracks.
Model-Based Infra
Model-based infra covers training and serving concerns. The core question is whether the compute stack can run the model change safely and efficiently.
Common checks include:
- Whether the proposed model architecture fits the available GPU memory
- Whether the serving engine supports the required inference pattern
- Whether runtime acceleration, batching, quantization, or adapter logic still works
- Whether multi-node or multi-server communication becomes a bottleneck
- Whether the change increases the risk of OOM, throttling, or poor GPU utilization
If a service uses multiple models, or multiple adapters attached to the same foundation model, the serving strategy becomes especially important. Distillation, compression, routing, and adapter loading can all affect latency and reliability.
Service-Based Infra
Service-based infra covers the distributed system around the model. The core question is whether the product service can keep its expected behavior after the model change.
Common checks include:
- Whether the API contract still matches the model’s input and output behavior
- Whether caching, queues, retries, rate limits, and timeouts still behave correctly
- Whether context management introduces surprising latency, cost, or security risk
- Whether logs and traces are sufficient for debugging failures
- Whether the service can scale under realistic request patterns
QA may own full end-to-end product validation, but MLE and MLOps teams should still validate these service-level risks before a model change reaches QA. The earlier the infra team catches incompatibility, the cheaper the iteration becomes.
In practice, the two metrics infra can influence most directly are latency and cost.
Latency Test Strategy
Latency is one of the first things users feel in an AI product. A model improvement that makes the service noticeably slower may not be a product improvement at all. At the dev stage, latency tests should isolate the model change from unrelated system noise as much as possible.
I usually think about latency tests across three layers: parallelism, input/output size, and service dependencies.
Parallelism
Parallelism tests ask how the system behaves as request volume increases. The exact scale depends on the business and product requirements, but the test levels should usually grow in a logarithmic pattern.
For a smaller service that calls a single API endpoint or inference engine, it may be enough to test concurrency levels such as 1 / 2 / 4 / 8 / 16 / 32. A service with millions of active users will need a much larger and more realistic load-testing plan.
The important metrics are not only average latency. The team should also track throughput, error rate, queueing time, timeout rate, p95 latency, and p99 latency.
Input and Output Size
Input/output size tests ask how latency changes as prompt length and generated output length change. The business task matters, but for this layer I care most about controlling token counts.
A useful matrix is:
| Input size | Output size | Purpose |
|---|---|---|
| Short | Short | Baseline fast-path latency |
| Short | Long | Generation-heavy latency |
| Long | Short | Context ingestion latency |
| Long | Long | Worst-case latency pressure |
The prompts should be designed so that input size and expected output size are stable across runs. The comparison environment should also be identical. Otherwise, the test may measure prompt variance or infra drift instead of the proposed model change.
Service Dependencies
Service-level latency also depends on everything around the model: preprocessing, retrieval, context construction, post-processing, parsing, buffering, warmup, cooldown, cleanup, retries, and external services.
These dependencies should be tested in a simulated environment that stays as close as possible across experiment runs. When possible, A/B testing is better than comparing today’s results with results generated a month ago, because the surrounding system may have changed in the meantime.
Cost Evaluation Strategy
Cost may be less urgent for some large companies, but it is critical for AI startups. As traffic grows, model inference can become one of the largest costs in the product. A proposed model change should not move to QA until the team understands its cost impact.
The main cost categories are:
- API cost: Third-party model providers usually charge by token usage. If one product request triggers several model calls, small prompt changes can produce large cost changes.
- GPU resources: Persistent and on-demand GPU clusters are expensive. Low utilization, memory fragmentation, inefficient batching, and I/O bottlenecks all turn compute into wasted budget.
- Operational cost: Engineering time, maintenance burden, monitoring complexity, and electricity costs can become meaningful, especially when the serving stack grows more complex.
A practical cost review should answer five questions:
- How many input and output tokens does each request consume?
- How much of that token usage is unnecessary for the user-facing task?
- What are the GPU utilization, memory utilization, and fragmentation patterns?
- How much engineering work is required to adopt and maintain the change?
- Does the expected product or research gain justify the added cost?
The final output should be a clear recommendation, not just a metric dump. Leadership should be able to see whether the change reduces cost, keeps cost roughly neutral, or introduces a cost risk that needs an explicit business decision.
Final Words
Improving the model iteration process does not stop here. After research validation and infra validation, the next stage is to look closely at prompt engineering from a product-development perspective. That discussion overlaps with my earlier prompt engineering whitebook, but it approaches the topic from a very different angle.
Before that blog comes out, Stay Hungry, Stay Foolish.