Where should AI run when it becomes business-critical?

In Part 1, I argued that the economics of AI change once a business becomes dependent on it.

That creates a second decision: where should those workloads run?

For most organisations, public AI services are the obvious place to begin. There is no infrastructure to purchase, capacity is immediately available, and new models can be adopted quickly.

That flexibility is valuable while an organisation is still experimenting.

Production workloads deserve a different conversation.

Successful pilots change the economics

There is a substantial difference between occasional AI use and a workload operating continuously across the business.

Once AI is processing a steady flow of documents, customer interactions or operational tasks, demand starts becoming predictable. At that point, the organisation can compare different ways of buying the same capability.

Public API consumption may remain the best option. Dedicated capacity may become more attractive for sustained workloads. Private infrastructure may suit workloads where data control, predictable performance or economics make it worthwhile.

The important point is that the architecture chosen for an experiment does not automatically deserve to become the permanent production architecture.

I expect many organisations will leave that decision too late, after workflows and integrations have already made the workload difficult to move.

Model capability should match the workload

Another assumption worth challenging is that every AI workload needs the most capable model available.

Many will not.

Complex reasoning may justify a frontier model, while a repetitive document processing or classification task may perform perfectly well on a smaller model.

At low volume, that difference might be insignificant. At scale, paying for capability the workload never uses can become expensive.

Model selection and infrastructure placement therefore belong in the same conversation. The workload should determine the level of capability required and the environment best suited to delivering it.

Control may matter more than cost

For some workloads, price will not be the deciding factor.

AI is increasingly being given access to customer information, intellectual property, source code, financial records and operational data. That brings familiar infrastructure concerns back into focus.

Organisations need to understand where processing occurs, how dependent a workload is on an external service, and whether it can be moved if the commercial or technical environment changes.

Where the model runs also determines where the business sends its information.

Before sending customer records, internal documents or commercially sensitive material to an AI service, the organisation needs to decide whether that data should leave its environment at all.

A provider’s commitment not to use customer data for training matters. It does not, by itself, answer where that data is processed or retained, who can access it, or which jurisdiction applies. Those assurances need to be understood in the context of the service being used and the controls the business requires.

For some workloads, a locally hosted model offers a more direct answer. When the full processing path stays within a controlled environment, sensitive information can remain inside the business’s own boundary.

That environment still needs to be secured and maintained. The difference is that the business can control the data path directly, rather than relying entirely on an external provider’s arrangements.

Performance can matter as well. An interactive service may require rapid responses, while a background task may care more about throughput and cost.

These are business continuity and control decisions. AI does not make them disappear.

Private infrastructure has its own economics

There is an equally important counterargument.

Running AI privately does not make the workload free.

Hardware needs to be utilised well enough to justify the investment. Power, support and lifecycle costs remain. The technology is evolving quickly, which creates a real risk of buying expensive infrastructure too early.

For many organisations, public AI services may remain the best economic and operational choice for a long time.

That is a perfectly reasonable outcome.

The problem begins when the original placement becomes permanent through inertia rather than a deliberate decision.

AI needs a workload strategy

Enterprise infrastructure has always worked best when workloads are placed according to their requirements. AI should be approached the same way.

Some workloads will continue to use public frontier services. Others may move toward private infrastructure, hosted inference or smaller specialised models as their requirements become clearer.

The decision should follow the economics of the workload, its data requirements, performance expectations and the organisation’s need to retain flexibility.

That approach gives the business something more valuable than a fixed position on cloud or on-premises infrastructure.

It gives the business options.

Once AI becomes business-critical, default placement is no longer good enough.

The next question is how private inference fits into that strategy, and whether businesses really need to own expensive GPU infrastructure before they can take advantage of it.

Nelson Lai

Next in the series: You do not need to own the GPUs to run private AI.

Previous: The real cost of AI starts when you can’t turn it off