Start with usage scenarios
Sizing begins with a description of uses: conversation, documentary research, extraction, long generation or agents. Each scenario mobilizes resources differently.
The number of concurrent users, peak periods, input size and expected response speed must then be estimated. A daily average often hides the constraints that actually determine the architecture.
Understand the role of GPU memory
GPU memory accommodates the model weights, the context-related cache, and the structures needed for inference. A larger model or a longer conversation increases this consumption.
Quantification reduces the footprint, with a potential compromise on quality. When the model exceeds the capacity of a GPU, it can be distributed, but this distribution adds network and coordination requirements.
Predict concurrency and latency
A configuration capable of responding to one user can slow down significantly as requests multiply. The inference engine combines some processing, but the capacity remains limited by memory and calculation throughput.
The objectives must distinguish the time before the first token, the generation speed and the total time. These metrics allow the architecture to be tested on representative loads rather than on a simple demonstration.
- Concurrent users at peak times
- Average and maximum length of contexts
- Number of tokens generated
- Latency target per usage
- Room for growth
Don't neglect storage and networking
Models, documents, vector indexes and logs occupy different volumes and do not have the same performance needs. Slow storage can delay model loading or knowledge ingestion.
The network becomes critical when a request crosses multiple services or when the model is distributed across multiple machines. The architecture must also provide for backups, restoration and separation of environments.
Define the level of availability
A platform used for a pilot does not require the same redundancy as an assistant integrated into a critical process. High availability, disaster recovery and non-disruptive maintenance naturally increase the number of components.
The service level must therefore be explicit. Oversizing all the bricks from the start consumes budget; ignoring availability creates a fragile dependency.
Test before final purchase
The estimates give a first envelope, but a test on the model and real queries remains the best validation. It allows you to measure memory, throughput and saturation points.
An evolving architecture plans from the start how to add capacity, separate new uses or change models. Sizing then becomes a trajectory rather than a fixed figure.