Sovereign AI is decided in the infrastructure, not in the model

Sovereign AI is a claim about control, not about origin

The term sovereign AI is used for two different things. One is industrial policy: national compute capacity, models trained in the region, supply chains that do not run through a single vendor. The other matters to any organisation buying or building something this year, and it is far more concrete. Sovereign AI in that sense means you can answer four questions about your own AI use without doing research first.

• Where does inference happen? Not where your application is hosted, but where the model actually computes the answer.
• Which model version answers? Pinned and documented, or whatever the provider shipped last night.
• Who may ask it, with which data? Access control and tenant separation, enforced by the platform rather than by convention.
• What leaves your environment, and what happens to it there? Prompts, uploaded reference material, generated results and logs.

A model built in Europe that you call through somebody else's API gives you none of these four answers. A model from anywhere in the world, running on hardware in an environment you control, gives you all of them. That is the reason this article is about infrastructure and not about model provenance: sovereignty is produced by the operating arrangement, not by the label on the weights.

This matters commercially, not just philosophically. Most organisations meet the question through a customer, a tender or an auditor asking where the data in an AI feature goes, and by then the answer has to exist. Retrofitting it means moving the place where inference runs, one of the more expensive changes you can make to a working application. The same logic applies one level up to the whole platform, which is the subject of digital sovereignty.

AI infrastructure is more than a stack of GPU servers

People talking about AI infrastructure usually mean compute alone. In practice there are five layers, and each of them can stall a project.

• Compute. GPU or specialised accelerator hardware, sized against model size, expected load and latency requirement. Whether it is rented, run in your own data centre or consumed from a sovereign cloud is a consequence of that sizing, not a decision to take first.
• Storage. Model weights, reference material, logs. Models are large, and they are held several times over once more than one version has to run in parallel.
• Network. Throughput and latency between application, inference and data sources. Tenant separation starts here, not in the application.
• Platform. The orchestration that starts, scales and distributes models and controls access. Usually a container platform, because GPU capacity can be allocated predictably through one.
• Operations. Monitoring, patch and update management, recovery, on-call duty. The layer that is missing in the pilot and decisive in production.

Technically this can be cut into three building blocks that should stay deliberately interchangeable: a hosting layer for open or licensed models, an inference runtime on the accelerator hardware, and an orchestration layer for scaling, load distribution and access control. Keep those three cleanly separated and a new model version, a different hardware backend or a change of hosting environment costs configuration rather than a rewrite.

Two things explicitly do not belong in that list. Training models is a separate discipline with a different load profile from inference, which is the use of a finished model to answer requests. And the data layer, meaning data integration, vector databases and retrieval pipelines, is a project of its own with its own owners. Packing both into the inference platform project means planning three projects as one.

AI security is not a property of the model

Security discussions about AI tend to be about the model: hallucinations, jailbreaks, filters. Those are real topics, but they describe how a system behaves, not the surface it presents to an attacker. That surface sits in the infrastructure, and it has four sides.

Data. With external model APIs, every request leaves your environment together with the data in the prompt. That is harmless while only public information travels, and delicate the moment personal data, trade secrets or regulated content are involved. In practice it is rarely documented what actually travels, because prompts are assembled from templates, context windows and attached documents.

Access. Who may query which model with which data? An inference environment without access control and without tenant separation is a shared database with a natural-language interface. Several applications on the same GPU need the same separation as several applications on the same database server.

Models. Without control over model versions you get behavioural changes you did not ask for. A vendor update can change answers that were already signed off inside a process. For audited workflows a pinned, traceable model version is a security requirement and not a matter of convenience.

Prompts and results. Prompts, uploaded reference material and generated results are themselves data worth protecting. Standard terms of use from external providers frequently permit reusing them for model improvement. Towards your own clients the opposite can only be proven when it is technically enforced.

The fifth side is availability. As soon as an AI function sits inside a customer process, its outage is an operational incident. From then on it needs recovery, on-call duty and a named owner, not just a dashboard. The general case for the rest of the platform is covered in cloud security best practices.

Free whitepaper · 8 pages

AI Engine

By Andreas Hankel, CTO onehundred. Download in exchange for your e-mail address, no sales call.

Get the whitepaper

AI and GDPR: it starts with where inference runs

Whether an AI application can be operated in line with the GDPR is decided at three points close to the infrastructure. The legal assessment belongs with your legal department or with specialised counsel. Technically, though, it can only be prepared this way.

Where processing happens. Processing takes place where inference runs, not where your application is hosted. If an application runs in Frankfurt and sends its requests to a model API in another jurisdiction, the place of processing is the model API. An inference environment in your own data centre or in a European sovereign data centre moves that point back under your control.

Who acts as a processor. Every service in the chain has to be named, from hosting through inference to logging and monitoring. With self-operated AI infrastructure that chain is short and documentable. With a chain of SaaS tool, middleware and model API it is often longer than the record of processing activities suggests.

Who has access. Access means not only your own staff but also support and operations accounts on the provider side, plus the question of whether prompts and results may be reused for training. This is where terms of use and privacy policy can diverge, and reading only one of them produces a confident wrong answer.

None of this amounts to a ban on external services. In practice it leads to a hybrid model: uncritical use cases keep running on standard tools, while sensitive or customer-related processing happens in a controlled environment. The precondition is a deliberate split. Without one, either the strictest case becomes the standard for everything, which blocks all use, or, more commonly, the loosest one does.

How the same questions are answered for the platform as a whole, beyond AI, is set out in the article on cloud compliance.

EU AI Act: risk classes, evidence and the moved deadlines

The EU AI Act classifies AI systems by risk. It applies to providers and deployers placing AI systems on the EU market, which includes organisations outside the Union that serve it. For infrastructure work one risk class matters above all: for high-risk systems the regulation requires a conformity assessment and documented evidence. Both reach directly into the architecture, because an organisation that does not know and does not log where its inference happens cannot produce any of it.

The deadlines have moved. Following the agreement on the Digital Omnibus package in June 2026, the high-risk obligations apply to standalone AI systems from 2 December 2027 and to systems embedded in products from August 2028. Anyone still planning against the originally scheduled date of August 2026 is working from a wrong picture of the timeline and has more room than they assume.

Two boundaries matter to us, because they get mixed up regularly. First: whether your specific system counts as high risk, which role you take under the regulation and which training obligations follow from that is a legal assessment. We do not make it, and statements about national implementation belong in legal advice as well. Second: technical evidence capability can be established independently of that, and it is the precondition for a legal assessment having anything to assess.

In concrete terms, technical evidence capability means this. It is documented which model in which version answers on which infrastructure in which location. It is logged which systems were allowed to send requests. It is traceable when any of that changed. In a self-operated inference environment that is a configuration question. With an external API it is a question of what the provider commits to and discloses.

For practice the conclusion is uncomfortable but simple: AI compliance is to a large extent infrastructure work, and it should be finished before the first productive application rather than after the first audit.

Make or buy: own GPU infrastructure against an API is a lifetime calculation

The question usually gets answered too early with a feeling and too late with numbers. It becomes defensible across three variables.

1. Expected request volume. How many requests will arrive over the next twelve months? Not in the best case, but in the likely one.
2. Duration of use. Is the use case permanent or a campaign? Own infrastructure amortises over time, not over a peak.
3. Evenness of load. Constant load keeps a GPU busy. Load in spikes leaves it idle most of the time and is exactly the profile an API was built for.

Behind this sits a simple cost structure. Self-hosted inference infrastructure has high fixed costs and low marginal costs, while external model APIs have no fixed costs and bill per request. At small to medium volumes APIs therefore still win more often than not. That is not an argument against your own AI infrastructure, it is an argument for doing the calculation honestly and allowing the outcome that the API stays cheaper.

Three items belong in the calculation that rarely show up in tool comparisons: the operating effort for your own platform, the maintenance of model versions, and counterparty risk. Price increases, changed rate limits or new terms of use from an API provider are not a hypothetical scenario, they are a cost factor with an unknown sign. Anyone comparing LLM costs over three years should work with ranges on both sides.

And there are cases where the calculation does not decide at all. If a use case processes personal data, trade secrets or regulated content that must not leave your environment, the sovereignty requirement overrides the cost calculation. The question is then no longer whether self-hosted inference is cheaper, but how small it can be built.

Self-hosted LLM: model, runtime, sizing

Running a self-hosted LLM is technically less spectacular than it sounds and operationally more demanding. Three decisions follow in order.

The model. Open models, for instance from the Llama or the Mistral family, can be run yourself and pinned to a version. Model choice is the most easily reversible decision in the whole undertaking, as long as the hosting layer is built to be interchangeable. For many tasks, such as classification, extraction or summarising internal documents, a smaller model answers well on considerably less hardware.

The inference runtime. It loads the model weights, holds them in the memory of the accelerator hardware and answers requests. Its metrics are latency per request, concurrent requests served and hardware utilisation. Those same three numbers later steer operations.

The sizing. It follows from model size, expected load and latency requirement, not from a hardware datasheet. The defensible order runs: fix the use case and the response times, derive the model size, determine the memory needed for the weights, add concurrency, and only then select hardware. Start from the other end and order a dedicated GPU server before the load is known, and you buy either too much or the wrong thing.

What gets underestimated is rarely the first deployment, it is everything after it: several model versions in parallel, a switch without downtime, clean separation of several applications on the same hardware, and a recovery path that also works at night. What that looks like as a service is described on the page about AI infrastructure.

Why AI initiatives fail in operations

The numbers are unpleasantly clear. S&P Global reports an abandonment rate of 42 per cent for AI initiatives in its "2025 Voice of the Enterprise" survey. IDC, together with Lenovo, puts the share of proofs of concept that are never rolled out at 88 per cent. The RAND Corporation arrives at more than 80 per cent failed AI projects. Depending on the study and the definition, then, between 42 per cent and more than 80 per cent of initiatives never reach production. The causes named are predominantly GPU capacity planning, data pipelines and a missing operating model, not the quality of the models.

The pattern behind it is always the same. A pilot runs on one machine with a data export and proves that the use case works. Production, however, needs orchestration, tenant separation, monitoring and recovery. That transition is rarely planned as a project with its own budget, and ownership often ends with the pilot.

A workable operating model for AI workloads answers four questions.

• Who gets called? On-call duty, an escalation path and defined availability, as for any other productive service.
• What gets measured? Besides latency and utilisation, the cost per request and the quality of answers, including drift, meaning the slow divergence of results from expected behaviour.
• How are models updated? Planned update management with a test run and a way back, instead of silent version changes.
• How does it scale? Capacity management driven by load, with a threshold at which a person decides, instead of an invoice that shows up at the month-end close.

That is the core of what gets described as MLOps: not a tool, but ownership, measurement and a repeatable path from a new model version into operations.

Start with an assessment, not with hardware

The defensible entry point is a technical AI readiness assessment. It answers three questions and is deliberately a platform question, not an evaluation of your use cases.

1. Data infrastructure. Are the data the use case needs available in a form a productive application can consume, or does only an export from the pilot exist so far?
2. Compute capacity. What capacity exists, what is needed, and is GPU capacity planned in your organisation or procured case by case?
3. Sovereignty. Which requests must not leave your environment, and where does that come from: contracts, regulation, or commitments made to your customers?

From the answers follows the make-or-buy calculation, and from that the target picture: your own data centre, a European sovereign data centre, a hybrid model, or staying on the API for now. Hardware and platform are selected only after that.

Two notes from practice. First, the question of where inference runs is easier to answer before data flows, tools and expectations have set. Asked only after the pilot, sovereignty becomes a retrofit, and retrofitting costs more than planning. Second, a use case where a customer has already asked "where does our data run?" is not a warning sign, it is the normal case of tomorrow.

Organisations that want to go this way without building the underlying infrastructure themselves will find in a private cloud the environment an inference platform subsequently runs in. The full argument, including the make-or-buy variables, is set out in the AI Engine whitepaper.