Reference Architecture for Sovereign AI: Five Layers, Flows, Threats and Evaluation

Nearly every AI vendor calls itself "sovereign" today. Some host your data in your region, some put a flag on their website, others promise a "trusted cloud". But a label is not an architecture. An AI is truly sovereign when you can answer one simple question, with evidence: who controls what, and what happens if the relationship ends tomorrow or the law changes?

This article is the synthesis of a series that started with what AI is, continued with AI and your documents and running models locally, and looked at data sovereignty. It is written for architects, CTOs and security leads: it describes a reference architecture component by component, with the trade-offs and failure modes I have met or consider likely. I have myself designed an on-premise LLM environment, with an open model from the Granite family served by SGLang on Kubernetes. I refer to it in generic terms only: no confidential detail, no figures from it. Numbers in this article are visible calculations or orders of magnitude flagged as illustrative.

What "sovereign" really means

Sovereignty is not a checkbox: it is a set of four forms of control. An AI can be strong on some and fragile on others, and the surprises hide in the gaps.

• Data sovereignty. You know where your documents, your questions and the AI's answers are, and who can access them. None of it is used to train a third party's model.
• Infrastructure sovereignty. The servers are in a place you choose, under a jurisdiction you know, at your site or with a local host.
• Model sovereignty. You can run the model yourself, replace it with another and evaluate it on your own cases. You do not depend on a single black box.
• Operational sovereignty. Your teams, or a partner you choose, know how to run the system. If a provider disappears, you keep control: access, documentation, the right to take over operations.

The local hosting trap

A server located in your country is not necessarily sovereign. If the provider operating it is subject to a foreign law, such as the CLOUD Act discussed in the article on data sovereignty, or if the model running on it can be neither inspected nor replaced, the dependency remains entire. Location is only one of the four dimensions.

A note on method: these four forms of control translate into testable properties. "I know where my data is" becomes an inventory of volumes and outbound flows. "I can replace the model" becomes a switchover test on a reference set. Everything below tries to make these properties verifiable rather than declarative.

The five layers of a controlled AI

An enterprise AI is not "a model". It is a stack of layers, each with its own role, interfaces and failure modes. Here they are, from top to bottom, with a cross-cutting block running through all of them.

1. Infrastructure

This is the physical base and its control plane: GPU-equipped nodes, storage, a network, hosted in a data center whose physical and legal access you control. The key questions are simple: where is it, who has access, what happens in an outage? Sizing depends on the number of concurrent users and the size of the models, as I detailed in the article on running AI locally.

Kubernetes as the foundation. Kubernetes is not mandatory, but it brings three useful things here: a reproducible declarative model (the same manifest on site and at a host), a scheduler that understands scarce resources, and an ecosystem for network control and workload identity. Its cost is real: a team that knows how to run it. For a single GPU server and a single model, a systemd service and a container are enough, and I do not advise installing a cluster for show.

GPU nodes and operator. A GPU node needs the driver, a suitable container runtime, the device plugin that exposes GPUs as a schedulable resource, hardware feature labeling and monitoring. According to its documentation, NVIDIA's GPU Operator automates this software on Kubernetes (driver, device plugin, container toolkit, labeling, DCGM metrics) and is deployed with Helm. The point is to avoid drift between nodes, a classic source of outages. Driver, CUDA, inference engine and kernel versions form a compatibility matrix: I pin it and test it at each update, never directly in production.

Scheduling. A pod asks for whole GPUs through the extended resource nvidia.com/gpu; the scheduler does not share a GPU that way. To share a GPU there are hardware-specific mechanisms (partitioning into isolated instances, time slicing), with isolation and latency trade-offs to verify for your GPU model. My reflex: one production model per dedicated GPU or GPU group, taints on GPU nodes so nothing else lands there, and explicit priorities so interactive inference is not evicted by a batch job.

# Illustrative excerpt: an inference deployment that requests 2 GPUs
# and can only land on the reserved GPU nodes.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-large-model
  namespace: inference
spec:
  replicas: 1
  selector:
    matchLabels: { app: llm-large-model }
  template:
    metadata:
      labels: { app: llm-large-model }
    spec:
      nodeSelector:
        pool: gpu-inference
      tolerations:
        - key: dedicated
          value: gpu
          effect: NoSchedule
      containers:
        - name: engine
          image: registry.internal/inference-engine@sha256:<digest>
          resources:
            limits:
              nvidia.com/gpu: 2

The image is referenced by digest rather than by a mutable tag: this is the first link of the supply chain, which I come back to below.

Storage. Three uses not to be confused: model weights (large read-only files loaded at startup), state data (vector store, PostgreSQL, logs) and engine caches. Weights live in an internal registry or object store, with a local copy on the nodes' NVMe disks so that startup does not depend on the network. State data goes on replicated, encrypted storage. A 70-billion-parameter model in 16 bits weighs about 140 GB (70e9 x 2 bytes): load time is not negligible and drives recovery time after an incident.

Network and zone segmentation. The principle: one zone per sensitivity level, and only explicitly listed flows pass between them. In Kubernetes, NetworkPolicy objects only take effect if the network plugin implements them, and by default a pod is not isolated: all traffic is allowed, according to the documentation. So you start from default deny, ingress and egress, in each namespace.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: inference
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: engine-from-gateway-only
  namespace: inference
spec:
  podSelector:
    matchLabels: { app: llm-large-model }
  policyTypes: [Ingress]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels: { kubernetes.io/metadata.name: gateway }
      ports:
        - port: 8000

Egress control. This is the most underestimated point. A local AI that can reach any address on the Internet is not isolated: a compromised pod, a booby-trapped container or a prompt injection that triggers a tool call will find a way out. I filter egress through a proxy or egress gateway with a domain-name allowlist, forbid direct external DNS resolution from sensitive zones (DNS is a classic exfiltration channel) and log every refused attempt. Standard NetworkPolicy reasons in IP addresses and ports; a domain-name allowlist needs an additional component, depending on the network plugin or service mesh in use. For a fully disconnected zone there is no egress at all, and updates come in through a controlled transfer process (see the supply chain below).

2. Models and the inference gateway

The model is the brain: an open model, evaluated on your own cases. The inference engine loads the weights into the GPUs and serves many users at once; vLLM and SGLang are two common examples. According to their documentation, both expose an HTTP server compatible with the OpenAI API (chat completions, completions, and other routes depending on the version), with their own extensions that the original API does not know. This compatibility is the cornerstone of reversibility, with one caveat: it covers most of the contract, not every parameter. Structured outputs, tool calling and reasoning modes vary between engines, and I test them explicitly before saying that one engine replaces another.

Sizing: KV cache memory. The limiting factor is not only the weight of the model but the key-value cache of each in-flight request. Its size per token is computed as:

bytes per token = 2 (K and V) x layers x KV heads x head dimension x bytes per value

Illustrative example, for a fictitious architecture with 32 layers, 8 KV heads, head dimension 128, 16-bit values: 2 x 32 x 8 x 128 x 2 = 131,072 bytes, i.e. 128 KiB per token. An 8,192-token context therefore takes 1 GiB per sequence, and 32 concurrent sequences 32 GiB, on top of the weights. This calculation, redone with your model's real parameters, tells you how many concurrent requests a GPU holds, well before raw compute does. Modern engines manage this cache in blocks and share it between requests with a common prefix, but the order of magnitude remains the right starting point.

The gateway. Between applications and engines, I place a gateway that exposes an OpenAI-compatible API and carries everything that should not be reimplemented in each application:

• authentication (OIDC token of the user or application) and per-identity quotas;
• routing between models, by logical alias (general-assistant, fast-summary, embeddings) rather than by real model name;
• policy enforcement (zone allowed for this data, tools permitted);
• writing the audit log and the metrics;
• fallback: if a model is saturated or unavailable, switch to another, if the policy allows.

Aliases are the key: the application asks for general-assistant, and the gateway decides, by configuration, which model and which version answer. Replacing a model no longer touches application code.

# Illustrative gateway configuration (generic schema, not a specific product)
models:
  general-assistant:
    target: large-model-engine.inference.svc:8000
    engine_name: large-model-v3
    allowed_zones: [confidential, internal]
    fallback: fast-summary
  fast-summary:
    target: small-model-engine.inference.svc:8000
    engine_name: small-model-v2
    allowed_zones: [confidential, internal, public]
  embeddings:
    target: embeddings-engine.inference.svc:8000
    engine_name: embeddings-v1
quotas:
  per_user: { requests_per_minute: 30, max_output_tokens: 2048 }

Model registry and versioning. A model is an artifact like any other: weights, tokenizer, chat template, generation parameters, license. I store them in an internal registry with an immutable version, file digests, the license (which may forbid some uses), provenance (where the weights come from) and the associated evaluation report. One detail that bites: some servers read by default the generation configuration shipped with the model, which can change sampling parameters without anyone asking, according to the vLLM documentation. I set these parameters explicitly in the deployment configuration.

Evaluation before switching. No change of model, quantization, template or engine version goes to production without rerunning the reference set (see the section on continuous evaluation). Quantization, for example, reduces memory and often speeds up throughput, but can degrade quality unevenly across languages: Arabic and legal text are good revealers, and I only trust my own measurement.

3. Knowledge and data

This layer gives the AI your context: the RAG method described in the article on AI and your documents. It includes ingestion (reading, chunking and vectorizing documents), a vector store (pgvector or Qdrant, for example) and above all access rights.

The golden rule has not changed: the AI must only retrieve what the user is allowed to read, and the filter applies at search time, not after generation. Filtering afterwards is too late: the confidential passage is already in the model's context, and nothing guarantees it will not influence the answer. Technically, each indexed chunk carries the metadata of its rights (groups, classification level, zone) copied from the source at ingestion, and the vector query runs with a filter derived from the user's identity:

# Principle: the rights filter is part of the vector query.
def search(question, user, k=8):
    vector = embeddings(question)
    filter_ = {
        "read_groups": {"any_of": user.groups},
        "classification": {"max": user.clearance},
        "zone": {"in": allowed_zones(user)},
    }
    return vector_store.search(vector, filter=filter_, limit=k)

Two pitfalls to watch. First, rights change: if a document becomes restricted or an employee changes department, the indexed metadata must follow, which requires incremental synchronization and a documented maximum delay. Second, filters applied in some approximate indexes can reduce the number of results returned or the recall quality when they are very selective; measure it on your engine rather than assuming. Finally, the vector index itself contains document fragments and is treated as data of the same sensitivity as the source (encryption, backup, zone).

4. Orchestration and agents

Between the question and the answer there is often more than one model call: rephrasing the question, searching several sources, querying a business system, checking a calculation, asking for human approval. Orchestration chains these steps. When the AI itself chooses actions (creating a ticket, sending an e-mail), it is called an agent, and the risk perimeter changes in nature: the model no longer only produces text, it triggers effects.

My design rules, without exception:

1. Per-role tool allowlist. A tool not declared for this role does not exist for the agent. Each tool has a strict argument schema, validated server-side, never model-side.
2. The user's rights, not the agent's. The tool runs with the identity of the person who asked (or a restricted delegated token), not with an all-powerful service account. Otherwise the agent bypasses the ACLs of the knowledge layer.
3. Human approval for the irreversible. External sending, writing to a system of record, deletion, payment: the agent prepares, the human confirms, and the confirmation screen shows what will actually be executed (arguments included), not the model's description.
4. Budgets. Maximum number of steps, tokens and tool calls per task, with a hard stop. An agent that loops burns GPUs and can multiply actions.
5. Tool-call logging: tool, arguments, identity, approval decision, result (truncated and masked). Without this log, an incident cannot be reconstructed.

This is the direct translation of the "excessive agency" category of the OWASP list of risks for LLM applications: too much functionality, too many permissions, too much autonomy. I always start read-only and add actions one by one.

5. Applications and interfaces

This is what your teams see: a chat window, an assistant embedded in document management, the CRM or the ERP, a mobile app. Two architectural points matter. Applications only talk to the gateway, never directly to an engine: that is what makes the lower layer interchangeable. And the interface must handle Arabic right-to-left, French and English, sometimes in the same conversation, including in the display of citations and document excerpts.

And everywhere: identity, policies, secrets, audit

The vertical bar in the diagram is not decorative. Without it, you have a demo, not a system. I break it down into six workstreams.

Identity. Single sign-on (SSO) via OpenID Connect backed by the company directory: the user presents a signed token, the gateway validates it and extracts roles (broad families: analyst, lawyer) and attributes (country, department, clearance). Roles suffice for tools; attributes are needed for documents ("files of country X"). Services authenticate to each other through workload identity and mutual TLS, not shared secrets.

Policy enforcement point. Decisions (this person, this data, this zone, this tool) are evaluated in one place, queryable and testable, rather than scattered in each application's code. The gateway is the enforcement point; the policy itself can be a dedicated engine or a simple versioned module. What matters: it is written in plain terms, reviewed, tested and deployed like code.

# Illustrative policy rules
rules:
  - name: confidential-stays-local
    if: { document_classification: confidential }
    then: { model_zones: [on-premise], network_egress: denied }
  - name: write-tools-need-approval
    if: { tool.type: write }
    then: { human_approval: required }
  - name: countries-isolated
    if: { document.country: "!= user.country" }
    then: { access: deny }

Secrets and keys. No secret in images or repositories: a secrets manager (or the orchestrator's secret store encrypted at rest, with a key you control), short-lived tokens, scheduled rotation, and above all an encryption key whose custody is clear. If the key sits with the provider that hosts you, encryption does not protect you from them.

Encryption. At rest (disks, database, backups, vector index) and in transit (TLS everywhere, including inside the cluster for sensitive zones). Encryption does not protect against a legitimate user asking a question they should not have been able to ask: that is the job of ACLs and policies.

Immutable audit log, with masking. Each interaction produces a structured record: timestamp, identity, model alias and actual version, zone, IDs of retrieved documents, tools called, policy decisions, token volumes, fingerprint of prompt and response. Append-only writing, on storage that forbids modification and deletion during the retention period (write-once storage, or hash chaining to detect any alteration). The raw text of prompts and responses contains personal data: I mask or pseudonymize it before writing to routine traces, and I only keep full content if a specific requirement justifies it, encrypted, with exceptional access logged too.

{
  "ts": "2026-03-04T09:12:41Z",
  "user": "u-5821",
  "role": "compliance",
  "alias": "general-assistant",
  "model": "large-model-v3",
  "zone": "on-premise",
  "documents": ["doc-1182#p4", "doc-0457#p2"],
  "tools": [],
  "policy": ["confidential-stays-local:ok"],
  "tokens": { "input": 1840, "output": 312 },
  "prompt_sha256": "<digest>",
  "masked_content": "What are the conditions of [PRODUCT] for [CLIENT_1]?",
  "chain": "<digest of the previous record>"
}

Observability. The metrics that matter for an inference service: time to first token (TTFT), throughput in tokens per second (per request and aggregate), queue depth, error and timeout rate, KV cache memory occupancy, GPU utilization, and cost per request. The latter is estimated simply: full hourly cost of the node (depreciation, energy, operations) divided by the number of requests served per hour at the target latency. TTFT degrades first under load (the queue grows), per-request throughput next (sequences share the GPU): I track both with percentiles, not averages. Same principles for RAG metrics: rate of empty retrievals, number of citations per answer.

Backup and recovery. What you back up: configuration (in a Git repository), model registry, document base, vector index (or, better, the source data, since the index can be rebuilt), audit logs. What counts is the restore test, with a written recovery objective: reload the weights, reinitialize the index, restart the engines. An incident that takes down a GPU node is first measured by the weight reload time (see above), hence the value of at least two replicas for critical services.

Reversibility. This is the property that makes sovereignty real. It is built through concrete choices: standard APIs between layers (OpenAI-compatible for inference, SQL and open formats for data), open model weights in common formats, no proprietary storage format (exporting documents, embeddings and logs must be a documented and tested exercise), versioned declarative configuration, and logical aliases for models. The test I recommend: once a year, actually replace the engine or the model on a test environment, and measure the time and the quality differences. A reversibility never tested is a hypothesis.

The journey of a question

To make these layers concrete, follow a question asked by an employee: "What are the early repayment conditions in our business contracts?"

1. Identify. The user is recognized through the company directory. Their role and attributes determine the document collections they can access.
2. Retrieve. The question is turned into a vector, and the search covers only authorized documents. The most relevant passages are selected.
3. Generate. The model, hosted on your side, receives the question and the passages, then writes an answer citing its sources.
4. Check and trace. Filters verify the answer (personal data, forbidden content, citations present), and the exchange is recorded in a protected audit log.

The diagram above is deliberately the path as the user sees it. Here is the same path as seen from the components, with the sequence of exchanges and the place where each decision is made.

Several choices are visible in this sequence. Policy is evaluated before any retrieval: a question aimed at a forbidden zone stops there. The gateway, not the client, calls retrieval, which prevents a modified client from choosing its own filters. The output check applies to the complete response or to windows of the stream, with an accepted trade-off: filtering the stream on the fly reduces perceived latency but makes filters harder to apply to a whole sentence. Writing to the audit is a step in its own right: if it fails, I prefer that the answer not be delivered in zones subject to a traceability obligation.

At no point does sensitive data leave the perimeter of its zone, and each step leaves a trace. This traceability, as much as location, is what convinces an auditor or a regulator.

Threat model

An enterprise AI has a particular attack surface, because the model processes instructions and untrusted data in the same stream. The OWASP list of the top ten risks for LLM applications (2025 edition) is a good reference framework; here are the threats that weigh most on this architecture, with what I set against them.

Two honest remarks. As far as I know, prompt injection has no complete defense at the model level: it is handled by architecture (limiting what the agent can do and what it can read), not by a magic filter. And masking by regular expressions or classifiers lets some cases through: it is a risk reduction, not a guarantee, and short retention compensates for its limits.

Supply chain, in practice. For a disconnected or highly constrained environment, the path is: download into an isolated import zone, verification of digests and provenance, scanning, copy into the internal registry, then deployment by digest. Nothing pulls an image or weights from the Internet at runtime. It is constraining, but it is what makes the term "on-premise" credible.

Where to run what: sorting by sensitivity

Sovereign does not mean "everything in-house". That would mean giving up the best models on the market for uses that are not sensitive at all. The most reasonable approach is to classify data, then route each category to the suitable place.

This sorting is only worth something if the system enforces it, not users' goodwill: each document's classification comes from the source (label, folder, metadata), and the gateway refuses to send confidential content to an alias that points outside the zone. A misclassified datum is the main weakness of this model. Hence a cautious rule: at ingestion, anything unclassified is treated as confidential.

Flows between zones are themselves an architecture decision. Here are the ones I allow and what I close.

The three main deployment modes each have their own logic, and none is superior in the absolute.

On cost, a calculation rule rather than a number: your own infrastructure becomes attractive when GPU utilization is high and stable, because an idle GPU costs as much as a saturated one. Occasional or highly variable usage favors pay-per-use. I always compare on the full cost per request, operations included.

One point of vigilance applies to all three: reversibility. Whatever the mode, you must be able to change host or model without rebuilding everything. That is why the layers are separated and linked by standard interfaces, and why it is better to avoid proprietary services that lock your data in a format you cannot export.

Continuous evaluation

I treat evaluation as a component of the architecture, on a par with the gateway. Without it, neither reversibility (you do not know whether the replacement is any good) nor security (you do not detect a behavior regression) can be verified.

The reference set. A few hundred representative cases, built with the business teams, in each language in use: question, expected passages, reference answer or success criteria. It also contains negative cases: questions with no answer in the documents (the right behavior is to say so), attempts to reach forbidden documents (the right behavior is to reveal nothing) and known injections. The set is versioned, and it evolves: each real incident adds a case.

What you measure. For retrieval: recall of the right passages among the top k. For generation: accuracy judged by experts (regular sample), share of answers with correct citations, rate of justified refusals. A model acting as automatic judge is useful for triage, but it has its own biases; I calibrate it against human judgments and do not delegate the final decision to it.

# Non-regression check before switching (principle)
def can_switch(candidate, reference, dataset, thresholds):
    r_ref = evaluate(reference, dataset)
    r_new = evaluate(candidate, dataset)
    for metric, threshold in thresholds.items():
        if r_new[metric] < r_ref[metric] - threshold:
            return False, metric
    # security cases: zero tolerance
    if r_new["acl_leaks"] > 0 or r_new["successful_injections"] > r_ref["successful_injections"]:
        return False, "security"
    return True, None

Progressive rollout means sending a small fraction of traffic to the candidate (or comparing it in parallel without exposing it), with immediate rollback by simply changing an alias in the gateway. The same tests are replayed at each change, not only at a model change: engine version, quantization, chat template, sampling parameters, embedding model (which itself requires a full reindex), and even a GPU driver update, since a change of compute kernel can slightly alter outputs.

The five-question test

Before signing with a "sovereign" AI provider, myself included, ask these five questions:

1. Where are my documents, my questions and the answers stored, and which jurisdiction does the provider fall under?
2. Who on your side can access my data, and is that access logged?
3. Is my data used to train another model?
4. Can I replace the model or the host without rewriting the application?
5. If we end our collaboration, what do we get back, in what form and within what time?

A provider confident in its architecture answers these five questions without detour, and in writing. For a technical audience, I add three checks: question 4 can be tested (ask for a demonstration of a model change), question 2 is verified in the audit log (ask to see a sample record), and question 5 is proven by a real export of test data.

A business case: a regional bank

Illustrative scenario

The case below is a scenario built to illustrate the approach. It does not describe a real client, and the indicators cited are those one would track, not results obtained. No performance figure is put forward.

The context

A retail bank operates in three countries in Africa and the Gulf. Its compliance, legal and customer relations teams spend hours searching through thousands of pages: internal procedures, regulator circulars, template contracts, case history. Some employees have started copying excerpts into consumer assistants, which is exactly what the risk department wants to avoid. The constraints are clear: client data must not leave the country, documents are in French, Arabic and English, and the regulator may ask for proof of who consulted what.

The chosen architecture

• Confidential data (client files, contracts): a Kubernetes cluster with GPU nodes in the bank's data center in each country, a mid-sized open model, possibly quantized after evaluation, and a local vector store. No direct egress to the Internet, imports by controlled transfer.
• Internal data (procedures, circulars, notes): the sovereign cloud of a local host, linked to the same identity system by OIDC federation, with encryption keys managed by the bank.
• Public content (brochures, FAQ, communication): an advanced cloud model, with no access to internal collections, called through the gateway like the others.
• One gateway per country, with common logical aliases, routing each request to the right zone according to the user's rights and the classification of documents. Separation by country is held by identity (a country attribute) and by the network.
• An audit log, append-only, for compliance, with masking of personal data in traces, and a procedure for exceptional access to full content, itself logged.
• An evaluation chain: a trilingual reference set built with the business teams, replayed at each change.

The choices that remain open in a scenario like this deserve to be named. The embedding model must handle Arabic and French correctly on banking vocabulary, which is measured on the reference set. Sharing the document base across countries is a legal question before it is a technical one. And the boundary between "internal" and "confidential" requires an arbitration by the risk department, which the architecture applies but does not decide.

The deployment, step by step

1. Scope. Choose a first use case with high volume and controlled risk (for example search in compliance procedures), classify the data, define success indicators and build, with the business teams, a reference set of questions and answers in the three languages.
2. Pilot. Set up the minimal foundation for a single service, compare two or three models on the reference set, measure, correct. This is when you discover what models are really worth in Arabic on your documents.
3. Industrialize. Strengthen security, plug in the directory, install monitoring, write the operating procedures, test restoration, train the teams. The pilot becomes a service.
4. Extend. Open to other departments and other countries, add agents on constrained tasks, and regularly reevaluate models as better ones arrive.

What you measure

An AI project is judged on indicators, not enthusiasm. For a case like this: average time to find a piece of information, share of correctly sourced answers, share of answers judged accurate by experts on the reference set, user satisfaction, service availability, TTFT at the 95th percentile and cost per request. You record the baseline before the pilot, otherwise you will never know what the AI changed.

The most common mistakes

• Starting with the model. It is the most visible choice, but access rights, document quality and security make the difference far more often.
• Neglecting document quality. Contradictory versions, unreadable scans, outdated procedures: the AI will find them, and cite the wrong ones. A cleanup is needed.
• Not evaluating. Without a reference question set, you can neither compare two models nor know whether an update improves or degrades the result.
• Filtering rights after generation. The confidential passage is already in the context. The filter belongs in the search query.
• Leaving network egress open. An "isolated" zone that can reach the Internet is not isolated.
• Giving the agent an all-powerful service account. The agent must inherit the user's rights, not bypass them.
• Wanting everything in-house. Keeping sensitive data local is wise. Putting public data there too is a needless expense.
• Forgetting operations. Updates, backups, monitoring, on-call: a sovereign system is a system someone operates.
• Giving agents too much power too early. Better to start with reading and search, then add actions one by one.

Key takeaways

• Sovereignty rests on four forms of control: data, infrastructure, model and operations. Hosting location is only one of them.
• Five layers (infrastructure, models, knowledge, orchestration, applications), crossed by identity, policies, secrets and audit.
• A gateway with a standard API and logical aliases makes the model replaceable; non-regression evaluation makes that replacement safe.
• Access rights apply at search time, and agents inherit the user's rights.
• The network is closed by default, egress as well as ingress; the supply chain is pinned and verified.
• Reversibility is the real guarantee, and it is tested.

Where to start? With three short deliverables: the data inventory with its classification, a reference set of a few dozen real questions, and a minimal gateway in front of a single engine, with identity and audit log from day one. The rest (agents, multi-zone, multi-country) is added afterwards on foundations that can already be measured. The architecture described here is not the only possible one, and the products cited are only examples: what must stay constant are the properties (rights at search time, closed egress, standard API, audit, evaluation, reversibility) and the ability to demonstrate them.

References

• vLLM: OpenAI-compatible server (official documentation, version as in the URL)
• SGLang: OpenAI-compatible APIs (official documentation)
• NVIDIA GPU Operator (official documentation, version 25.10)
• Kubernetes: Network Policies (official documentation)
• OWASP Top 10 for LLM Applications (2025 edition)