From Paying for Tokens to Investing in AI Infrastructure

For the past few years, the AI industry has conditioned us to think in tokens.

Every prompt has a price. Every conversation consumes credits. Every agent, workflow, or document ingestion job ultimately turns into another line item on the monthly invoice. It's become such a familiar model that many organizations have started treating token usage as the primary metric for evaluating AI.

Last week, Palantir CEO Alex Karp challenged that assumption. In an interview discussing the state of enterprise AI, he argued that companies have become far too focused on buying tokens instead of buying outcomes. His criticism wasn't directed at large language models themselves, but at an ecosystem where enterprises spend millions on AI consumption while struggling to demonstrate proportional business value.

Whether you agree with his perspective or not, his comments highlight a much larger shift that's already happening inside enterprises. As AI moves from experimentation to becoming part of day-to-day operations, organizations are beginning to ask a different question.

Not "How many tokens did we use?"

But rather:

"What did AI actually help us accomplish?"

That distinction may seem subtle, but it fundamentally changes how enterprise AI should be built.

AI shouldn't behave like a utility bill

Consumption-based pricing made perfect sense during the first wave of generative AI.

Developers needed a simple way to experiment. Companies wanted instant access to increasingly capable models without investing in infrastructure. Paying only for what you used was an easy proposition.

The problem is that successful AI projects rarely stay small.

A chatbot that begins as a proof of concept quickly becomes an internal assistant used by hundreds of employees. Teams start uploading documentation, developers integrate AI into applications through APIs, agents begin executing workflows, and suddenly AI is supporting critical business processes across the organization.

At that point, usage no longer grows linearly—it compounds.

More users generate more conversations. More automation creates more background processing. Larger knowledge bases require more retrieval and reasoning. Costs become increasingly difficult to predict because every new use case translates into more tokens being consumed.

Ironically, success becomes expensive.

That's an unusual incentive for enterprise software.

No one expects their database bill to increase because employees ran more SQL queries this month. Nobody worries that using Kubernetes more often will dramatically increase operational costs. These platforms are considered infrastructure: organizations invest in them because they enable the business, not because they meter every interaction.

AI is rapidly becoming the same kind of foundational technology.

The invoice isn't the only cost

Token pricing is only one part of the equation.

For many enterprises, the larger concern isn't how much each request costs, but where those requests are being processed in the first place.

Every prompt sent to an external provider raises operational questions. Where is the data handled? Which models process it? What governance policies apply? Can sensitive information leave the organization's environment? How does this fit with regulatory requirements or internal security policies?

Cloud providers have invested heavily in answering these questions, and many organizations are comfortable with the safeguards they offer. Others—particularly those operating in highly regulated industries or managing sensitive intellectual property—prefer a simpler answer altogether: keep the entire AI stack inside their own infrastructure.

That decision isn't just about compliance. It changes the economics of AI.

When organizations own the infrastructure running their models, costs become significantly more predictable. Instead of paying for every prompt, they invest in compute capacity that can be shared across teams, applications, and workloads. More importantly, they gain complete control over where data is processed, how models are deployed, and how AI evolves inside the organization.

Owning the infrastructure changes what you optimize

When every interaction has a direct monetary cost, optimization usually means reducing usage.

Prompt engineering becomes an exercise in minimizing tokens. Teams think twice before enabling new use cases. Developers worry about the cost implications of adding autonomous agents or processing large document collections.

The conversation revolves around spending less.

Running AI on-premise creates a different set of incentives.

Once the infrastructure is already in place, the question is no longer how to consume fewer tokens. Instead, it becomes how to make better use of the compute resources you already own.

That's a much healthier optimization problem.

Rather than limiting adoption, organizations can focus on making their infrastructure smarter. A lightweight model can answer routine questions while larger models handle complex reasoning. Interactive chat sessions can be prioritized over background ingestion jobs. APIs, agents, document processing, and employee assistants can all share the same inference cluster while being orchestrated according to business priorities instead of competing equally for resources.

Capabilities like intelligent model routing, AI gateways, and workload prioritization make this possible. Rather than treating every request the same, the platform can determine which model is most appropriate, how resources should be allocated, and which workloads deserve priority. These architectural decisions improve both user experience and infrastructure efficiency without requiring employees to think about the underlying complexity.

Instead of optimizing invoices, you're optimizing your own platform.

AI infrastructure should work like enterprise infrastructure

This is where the conversation shifts away from models and toward architecture.

The model itself is only one component of an enterprise AI platform. What increasingly determines success is everything surrounding it: model routing, gateways, workload orchestration, security, governance, observability, and resource management.

These are the capabilities that allow organizations to scale AI without scaling costs at the same rate.

They also make AI feel less like an external service and more like any other piece of enterprise infrastructure.

Your storage platform isn't priced according to how often employees open files. Your identity provider doesn't charge per authentication decision. Your virtualization platform isn't billed based on every process it schedules.

They're platforms that exist to support the business efficiently.

As AI becomes equally foundational, enterprises are beginning to expect the same characteristics: predictable operating costs, operational control, governance, and the flexibility to deploy new use cases without worrying that every successful project will produce another surprise invoice.

This is also why open-weight models continue gaining traction in enterprise environments. They give organizations the freedom to choose where AI runs, avoid vendor lock-in, and optimize infrastructure according to their own operational and business priorities rather than the pricing model of a single provider.

The conversation is bigger than tokens

Alex Karp's comments sparked debate because they challenged one of the assumptions that has defined the generative AI market over the last few years.

But the more interesting takeaway isn't whether token pricing is good or bad.

It's that enterprise AI is maturing.

Organizations are no longer evaluating AI solely by the quality of its responses. They're evaluating it the same way they evaluate every other strategic technology investment: by the business value it creates, the operational control it provides, and whether its economics remain sustainable as adoption grows.

At Zylon, this is precisely why we focus on building a private AI platform rather than simply providing access to language models. AI gateways, workload prioritization, model routing, and on-premise deployment are not isolated features; together, they form the foundation of an AI platform that organizations can govern, scale, and optimize over time.

Ultimately, enterprises are not trying to maximize token consumption.

They're trying to make AI a reliable part of how their business operates.

As AI becomes a long-term strategic capability rather than an experimental tool, the conversation will increasingly move away from the cost of individual prompts and toward the value created by the infrastructure behind them.

Sources

• Alexander, Reed. "Palantir CEO Alex Karp says AI companies charging by the token are creating 'a cult.'" Business Insider, July 2026. https://www.businessinsider.com/alexander-karp-criticizes-ai-companies-token-costs-2026-7

• Berkowitz, Ben. "Palantir's Karp says AI customers should pay for outcomes, not tokens." Axios, July 2026. https://www.axios.com/2026/07/02/karp-palintir-openai-anthropic-amodei
  
  

Author: Daniel Gallego Vico, PhD, Co-Founder & Co-CEO at Zylon
Published: July 2026
Daniel specializes in secure enterprise AI architecture, overseeing on-premise LLM infrastructure, data governance, and scalable AI systems for regulated sectors including finance, healthcare, and defense.