Choose hardware around the model and the service you need to deliver. Include runtime memory, concurrency, power, cooling, software, and support in the plan.
Write down what the system needs to do.
A single-user knowledge assistant and a service answering many simultaneous requests have different requirements. Start with the intended models, expected users, input lengths, output lengths, and acceptable response times. Include any image, audio, embedding, or reranking workloads.
Model weights are only part of the memory budget. Runtime overhead and attention-cache memory also matter, especially as context length and concurrent requests grow. Quantisation can change memory needs and output quality, so evaluate the actual serving configuration rather than a model name alone.
Buy a compatible system, with a clear specification.
The accelerator, server, interconnects, storage, and network need to work together. Form factor and cooling requirements can rule out a simple card upgrade. Larger platforms may need substantial changes to racks, power distribution, and facility cooling.
Ask the supplier to confirm the exact bill of materials, condition, warranty, support boundary, destination eligibility, and lead time. A product-family name is not a full specification, and an indicative catalogue listing is not a stock allocation.
- Confirm the exact accelerator and supported server platform.
- Check power, cooling, space, and network readiness.
- Agree who installs, commissions, and supports each part.
Plan the software and data flows as carefully as the hardware.
A local inference deployment can include supported GPU drivers, CUDA, containers, an inference engine, authentication, model storage, and monitoring. Tools such as vLLM, Ollama, or NVIDIA NIM serve different operational needs and have their own compatibility and licence requirements.
Local model serving does not automatically keep every data flow onsite. Review retrieval sources, tool calls, telemetry, backups, and any connected cloud services. Document the data paths and test the access controls that protect the application.
Validate the workload and prepare the handover.
Use representative requests to check latency, throughput, memory headroom, and output quality. Include the expected concurrency and context lengths, then agree what a passing result looks like. Avoid treating vendor benchmark figures as a promise for a different workload.
Account for electricity, hosting space, support, maintenance, model updates, and staff time alongside the hardware price. A clear operating owner, recovery procedure, and capacity plan help turn a hardware purchase into a service the business can depend on.