Local AI: sizing memory, choosing an engine, deploying

Every request sent to a hosted AI service leaves your network, crosses the Internet and ends up in a data center whose location and retention policy you do not control. For a contract, a patient record or source code, that raises confidentiality questions. It also raises availability questions (what happens when the connection drops?) and cost questions (what does the bill look like once the whole organization is using it?).

Running a model on your own machines is possible, and the question is no longer "is it feasible" but "which machine, which format, which engine, for which load". This article is the sizing guide I wish I had had at the start: formulas you can redo by hand, orders of magnitude labelled as such, commands to adapt to your version, and the limits. The numerical examples use generic configurations and are illustrative: no performance measurement is claimed. Always measure on your own hardware and your own requests.

Where does the AI you use actually run?

A "cloud" AI runs on a provider's machines: you send text, you get text back. A "local" AI runs on machines you control: a laptop, a server in your premises, or a host in your jurisdiction. The relevant criterion is not geography but control: who sees the prompts, who keeps the logs, who decides on updates.

The diagram does not say one mode is better. It says where the data goes. The rest of the article is about what it costs, in memory, latency and operations, to bring that point of passage in-house.

A model is a file of weights

Once trained, a language model boils down to tensors of numbers (the weights, or parameters) and a description of the architecture. For the basics of neural networks, see AI explained simply. What matters here: this file can be copied. Training takes thousands of accelerators for weeks; inference takes loading the weights into memory and chaining matrix multiplications. The cost gap between the two is huge, and it is what makes local inference thinkable.

You still need to be able to get the weights. Top-tier proprietary models are only reachable through an API. Alongside them, many open-weight model families are published: Llama (Meta), Mistral, Qwen (Alibaba), Gemma (Google), DeepSeek, Falcon (Abu Dhabi), or Jais, designed first for Arabic. Each family comes in several sizes.

Two precautions. First, "open weights" is not "open source" in the strict sense: the license may restrict commercial use, impose user thresholds or acceptable-use clauses, and the training data is generally not published. Read the license of the exact version you deploy. Second, a downloaded file is untrusted code and data: prefer formats that do not allow arbitrary execution (safetensors, GGUF) and check provenance.

Sizing memory

Accelerator memory, not raw compute, is what first decides what can run. It breaks down into three items.

Weights

The formula is straightforward:

weight_memory ≈ number_of_parameters × bytes_per_weight

An 8-billion-parameter model in 16 bits (2 bytes) takes about 16 GB; in 8 bits, about 8 GB; in 4 bits, a bit over 4 GB, because quantized formats also store scale factors (on the order of 4.5 to 5 effective bits per weight for common 4-bit formats). A 70-billion model takes about 140 GB in 16 bits and roughly 40 GB in 4 bits. Here, 1 GB is 10^9 bytes.

The KV cache

During generation, each attention layer needs the keys (K) and values (V) of all previous tokens. The engine keeps them in memory instead of recomputing them: this is the KV cache. Its size is:

kv_memory = 2 × layers × kv_heads × head_dim × bytes × tokens × sequences

The factor 2 counts K and V. "KV heads" are those of grouped-query attention (GQA), where several query heads share one key and value head, which shrinks the cache considerably. The exact values are in the model configuration (the config.json file for a Hugging Face model). Take two generic configurations, with a 16-bit cache:

The last row shows why GQA exists: without it, a single 8,192-token sequence would cost about 21 GB of cache in the 70-billion configuration. With 8 KV heads, it costs about 2.7 GB.

Illustrative totals, activations and margin not counted:

Two lessons. For a single user, the weights dominate. For a shared service, the KV cache quickly exceeds the weights: the 32-sequence row does not fit on two 80 GB GPUs, whereas the same load with an 8-bit cache does, with a margin that still has to be reserved for activations. Several engines can quantize the KV cache (for example to FP8 or 8 bits); the effect on quality depends on the model and the context length, so it has to be evaluated. A model's advertised maximum context is not free: doubling the length will, at worst, double the cache per sequence.

Activations and buffers (compute graphs, kernel workspaces, fragmentation) are a smaller item, but never zero. That is why engines expose a memory usage cap (see below) instead of allocating the whole GPU.

Quantization: fewer bytes per weight

Quantization replaces 16-bit weights with shorter representations. The image-compression intuition is right up to one detail: the goal is not only to save space but to read fewer bytes for every generated token, which also speeds up decoding (see the next section).

Format families

• FP16 / BF16: the reference. BF16 keeps the dynamic range of FP32 and is what most official releases ship.
• FP8 and INT8: one byte per weight. FP8 brings a native compute gain only on recent GPUs that support it; on others it only reduces memory. Check against your accelerator and engine.
• 4 bits: the usual starting point for large models on constrained hardware. Three ecosystems dominate: GGUF k-quants (llama.cpp, Ollama; variants named for example Q4_K_M, Q5_K_M, Q6_K, Q8_0), AWQ and GPTQ.

AWQ (Lin et al.) picks the weights to protect based on activations observed on calibration data, rather than on their magnitude alone. GPTQ (Frantar et al.) quantizes layer by layer, minimizing reconstruction error. GGUF k-quants are block-wise quantizations, with scales stored per block; they work on CPU as well as GPU and let you offload some layers to the GPU when memory is short.

Why quality does not degrade uniformly

It would be comforting to believe in a loss of a few percent, everywhere. That is not what one observes in practice, for reasons that are partly mechanical:

• Rounding errors accumulate on long tasks: multi-step reasoning, a calculation or a program requires every step to stay exact, whereas a summary tolerates some lexical imprecision.
• Small models have less redundancy and tolerate removed bits less well. At equal memory, a larger and more heavily quantized model often beats a smaller and lightly quantized one (an empirical rule, not a law, to be checked on your tasks).
• Calibration data, when the format uses any (AWQ, GPTQ, and some GGUF variants with an importance matrix), is often predominantly English. My hypothesis, for which I have no general measurement, is that under-represented languages and domains (dialectal Arabic, local legal or medical vocabulary) are more exposed to degradation. This is exactly the kind of effect a test on your own texts detects and an English-language benchmark does not see.
• Perplexity, often cited to compare quantizations, measures next-token prediction, not task success: two quantizations with similar perplexity can behave differently on structured extraction or tool calling.

The practical rule: start from an 8-bit, or a higher-quality 4-bit (for example a Q5 or Q6 k-quant) if memory allows, then go down, and compare on your test set every time (see below).

Latency, throughput and memory bandwidth

Two distinct usage metrics:

• TTFT (time to first token): the delay between sending the request and receiving the first token. It depends on prefill.
• Generation throughput: tokens per second after the first, on the user side (perceived speed) or the server side (sum over all requests).

Prefill: compute-bound

Processing a prompt of N tokens costs roughly 2 x parameters x N floating-point operations (an order-of-magnitude rule for a dense model). For an 8-billion model and a 4,000-token prompt, that is about 6.4 x 10^13 operations: with, by assumption, 100 TFLOP/s sustained, on the order of half a second. Prefill uses compute units well, because all prompt tokens are processed together. A long prompt (RAG, conversation history) therefore directly lengthens TTFT.

Decoding: memory-bandwidth-bound

To generate one token, all active weights (and the sequence's KV cache) must be read from memory, for very little compute per byte read. A theoretical ceiling for a single sequence is therefore:

tokens_per_second ≲ memory_bandwidth / bytes_read_per_token

Illustration: a 4.5 GB quantized model on 100 GB/s memory tops out around 22 tokens per second; on memory ten times faster, around 220. Reality is lower (imperfect kernels, KV cache, engine overhead), but the order of magnitude explains two practical facts. Quantization speeds up decoding, because it reduces the bytes read. And two machines with comparable compute can have very different generation speeds depending on their memory.

For an interactive user, a few tens of tokens per second are enough to outpace reading speed. The real problem is simultaneous load.

Why batching changes everything

In decoding, reading the weights once can serve the next token of several sequences at once: the memory cost is nearly shared, the compute adds up. Total throughput therefore grows almost linearly with the number of sequences, until compute or reading the KV caches becomes the bottleneck. The price is a per-request latency that rises with load. The whole art of an inference server is to maximize this throughput without degrading latency beyond what your service tolerates.

Mixture of experts: total versus active parameters

Some models are mixture of experts (MoE): for each token, a router activates only a fraction of the feed-forward blocks (the "experts"). The distinction is between total and active parameters. Hypothetical example: a model with 100 billion total parameters of which 10 billion are active per token.

• Memory follows the total parameters: all experts must be loaded, because you do not know in advance which will be called. This hypothetical model therefore needs memory in the 100-billion class.
• Compute and the bytes read for a single token follow the active parameters: at equal bandwidth, decoding close to that of a 10-billion model.

This advantage erodes with batching. With many simultaneous sequences, different tokens call on different experts, and almost all experts end up being read at each step. An MoE is therefore especially attractive when you have a lot of memory (large unified memory, multi-GPU server) but little bandwidth per token, and less so when memory is the constraint. Parallelism across several GPUs (expert or tensor parallelism) also adds communication costs.

Engines: single user or shared server

Two usage families call for different engines.

Individual or small group. llama.cpp runs GGUF models on CPU, GPU or both, with a light installation. Ollama wraps an engine from the llama.cpp family (originally) with a model catalog, a local API and memory-loading management. LM Studio and Jan offer a graphical interface. The goal is simplicity and hardware flexibility, not maximum throughput.

Shared service. vLLM and SGLang target throughput under concurrent load. Two mechanisms explain their appeal:

• Continuous batching: instead of waiting for a whole batch of requests to finish, the engine inserts new requests into the batch as soon as a sequence ends, token by token. The GPU stays busy even when requests have very different output lengths.
• Paged KV cache (PagedAttention, introduced with vLLM by Kwon et al.): the cache is no longer allocated as one contiguous block of maximum size, but in fixed-size blocks referenced by a per-sequence table, like virtual memory. This reduces waste from fragmentation and allows blocks to be shared between sequences. SGLang adds, with RadixAttention, reuse of common prefixes (system prompt, shared documents) across requests. The gain depends heavily on the share of common prefixes in your traffic: to be measured, not assumed.

llama.cpp also offers continuous batching and several parallel slots in its server, which makes it usable for a few concurrent users. The gap with vLLM or SGLang widens as concurrency and contexts grow.

A few commands (check them against your version)

Options change from one version to the next: check with the built-in help (--help) and the documentation of the installed version. Model names below are examples or placeholders.

Ollama, local use:

# Download and open an interactive session
ollama run llama3.1:8b

# See loaded models and where they run (PROCESSOR column: GPU, CPU, or split)
ollama ps

# Call the OpenAI-compatible API exposed locally (port 11434 by default)
curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "llama3.1:8b", "messages": [{"role": "user", "content": "Hello"}]}'

According to Ollama's documentation, the server listens by default on 127.0.0.1:11434, keeps a model in memory for 5 minutes after the last request (OLLAMA_KEEP_ALIVE), handles one request at a time per model by default (OLLAMA_NUM_PARALLEL) and queues up to 512 requests (OLLAMA_MAX_QUEUE). These defaults have varied across versions. Exposing the server on the network with OLLAMA_HOST=0.0.0.0 adds no authentication: put it behind a proxy that provides one.

llama.cpp, HTTP server:

# -ngl: layers offloaded to the GPU, -c: context, -np: parallel slots
llama-server -m ./model-q4_k_m.gguf -ngl 99 -c 16384 -np 2 --port 8080

The server exposes /v1/chat/completions, among others. According to the documentation, the requested context is in principle shared among the parallel slots: check this behavior for your version before sizing -c.

vLLM, OpenAI-compatible API:

# GPU memory usage cap and maximum context length
vllm serve <organization>/<model> \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.90 \
  --api-key "$VLLM_API_KEY"

The server listens on port 8000 by default and exposes /v1/chat/completions as well as a /metrics endpoint in Prometheus format. The --max-model-len option bounds the cache memory per sequence; --gpu-memory-utilization sets the fraction of GPU memory the engine may use; what remains after the weights and activations serves the KV cache. Confirm option names with vllm serve --help.

Hardware tiers

Laptop. For individual use: writing assistance, summaries, low-complexity code. A 1 to 8 billion model in 4 bits fits in a few gigabytes. Unified-memory machines (Macs with Apple silicon, some recent x86 machines) share memory between CPU and GPU and can load larger models than a dedicated graphics card of the same price would allow, with bandwidth remaining the limiting factor in decoding. Concurrency is very low: think one user.

Desktop with a graphics card. The card's video memory sets the ceiling: a medium quantized model plus a reasonable context. If the weights do not fit entirely, llama.cpp can offload part of the layers to the GPU and leave the rest on the CPU, at the price of a markedly lower speed because every token goes through the slow part. Suits a small team with low concurrency, or a developer workstation.

GPU server. Here sizing is about concurrency: how many simultaneous sequences, what context length, what acceptable TTFT. The weights determine the minimum number of GPUs; the KV cache determines how many sequences fit on top (see the table above). Beyond one GPU, tensor parallelism splits the weights but needs fast interconnects between cards; on cards linked only by the PCIe bus, communication can become the bottleneck. Also plan for power, cooling and card availability on your market, which is not always guaranteed.

A shared inference server

Going from a workstation to an organization is not just changing machines: you add layers around the engine.

Applications talk to an OpenAI-compatible API, which lets you change engine or model without rewriting clients and fall back to a hosted service if needed. The gateway (a proxy or an existing API gateway) carries authentication, per-team quotas, rate limiting and logging. Do not leave an inference engine directly exposed: most offer only minimal shared-key authentication and no fine-grained identity management. To give the assistant access to your documents, see AI and your documents: retrieval sits in front of the engine, which only receives context already filtered by access rights.

Monitoring

An inference server is monitored like any service, with its own metrics. Track first:

• TTFT and inter-token latency, as percentiles (p50, p95, p99) rather than averages: users notice the tails of the distribution.
• Running and waiting requests: a growing queue signals saturation before users complain.
• KV cache occupancy: close to 100%, the engine must evict or refuse sequences and latency degrades.
• Tokens per second, aggregated, in and out, to follow load and cost.
• GPU memory and temperature, hardware errors, and HTTP error rate.

vLLM exposes this information in Prometheus format on /metrics (for example metrics named vllm:time_to_first_token_seconds or vllm:num_requests_running at the time of writing; names evolve across versions, as does the one for KV cache occupancy). Combine them with GPU metrics (for example NVIDIA's DCGM exporter, if you are on that hardware) and a per-team dashboard.

Evaluate before choosing

Public model leaderboards help shortlist, not decide. Your traffic has its languages, formats, documents and accuracy requirements. A minimal evaluation fits in a day:

1. Gather 50 to 200 representative, anonymized requests, with an expected answer or a verifiable success criterion (field extraction, classification, correct citation, code that passes tests).
2. Run each candidate (model, quantization, context size) on this set with the same generation settings, preferably at low temperature for repeatability.
3. Score automatically what can be, have a human review the rest, and split the results by language and task type.
4. Measure TTFT and throughput in parallel under the real expected concurrency, not on an isolated request.
5. Replay the same set at every change of model, format or engine version: it is your regression test.

If you use a judge model for scoring, know that it has its own biases (preference for longer answers, for its own style) and validate it on a hand-reviewed sample. The test will often say that a smaller, well-quantized model is enough for a specific task: it is the best way to reduce the hardware you need.

Cloud or local: the comparison

Total cost

Comparing a subscription to the price of a graphics card is a classic mistake. The cost of local inference includes:

• the hardware, amortized over its realistic useful life (GPUs and models move fast);
• energy and cooling, proportional to power draw and utilization;
• operations: installation, engine and model updates, monitoring, on-call, security, and the engineering time this represents;
• hosting: space, network, redundancy if the service is critical.

The calculation that matters is the cost per million tokens actually served:

cost_per_million_tokens ≈ (amortized hardware + energy + operations) / tokens_served_over_the_period × 10^6

The denominator is the weak point: a GPU used at 10% costs ten times more per token than a GPU used at 100%. API prices change, and so do hardware prices: I deliberately give no figure. Do the calculation with your quotes, your electricity price and your measured traffic, and compare it to the per-token price of the API you would replace.

When local makes no sense

• Low volume. If your use is occasional, the amortization of a server will never pay off, and an API (with a contract suited to the data processed) will cost less.
• You need the best model. For long reasoning, complex agentic tasks or evaluations where the quality gap shows, open models often remain a notch below the best proprietary ones. Measure that gap on your cases; it varies.
• No team to run it. A server without monitoring or updates quickly becomes debt, and a security risk.
• Highly variable load. Capacity sized for the peak sits idle the rest of the time, whereas a pay-per-use service absorbs the elasticity.
• Local does not mean secure. A server in your walls must be protected like any sensitive system: network segmentation, authentication, logs, backups, updates. A local API without authentication is a potential leak.

The reverse exists too: data that legally or contractually cannot leave, offline operation, or a high and stable volume often justify local. The hybrid approach, which keeps sensitive data local and delegates the rest to a hosted service, is common; it ties into the question of data sovereignty.

What to take away

In short

• Weight memory ≈ parameters x bytes per weight. KV cache memory = 2 x layers x KV heads x head dimension x bytes x tokens x sequences. The first item is fixed, the second grows with load.
• Quantization reduces memory and speeds up decoding, but the quality loss is not uniform: it depends on the task, the model size and the language. Measure.
• Prefill is compute-bound, decoding is memory-bandwidth-bound. Continuous batching and a paged KV cache make servers like vLLM and SGLang suited to concurrency; llama.cpp and Ollama target individual use.
• An MoE needs the memory of its total parameters but only computes its active parameters, with an advantage that erodes under heavy concurrency.
• Monitor TTFT, queues and KV cache occupancy. Evaluate on your own cases before choosing. Compute the cost per token served, not the price of a card.

To start: install Ollama or llama.cpp on a workstation, load a quantized 8-billion model, and run your own requests against it with a small test set. In a day you will know whether local is enough for you, and which model and context size justify dedicated hardware. What follows are architecture questions: gateway, document access, monitoring. They are covered in the other articles linked above.

References

• Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention: https://arxiv.org/abs/2309.06180
• Zheng et al., SGLang: Efficient Execution of Structured Language Model Programs: https://arxiv.org/abs/2312.07104
• Lin et al., AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration: https://arxiv.org/abs/2306.00978
• Frantar et al., GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers: https://arxiv.org/abs/2210.17323
• llama.cpp server documentation: https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md
• vLLM documentation, online serving: https://docs.vllm.ai/en/latest/serving/online_serving/
• vLLM documentation, metrics: https://docs.vllm.ai/en/latest/design/metrics/
• Ollama FAQ: https://docs.ollama.com/faq