Introducing PrivateGPT 1.0: The Open Source Application Backend for Private AI

Some open source projects are interesting experiments.

Others become the proof that a problem is real.

When we published the original PrivateGPT in 2023, it hit GitHub's number one trending spot almost immediately. It was the first implementation that let you run a complete RAG pipeline offline, with no cloud dependency and no data leaving your machine. At the time, even the CEO of Langchain had publicly said that running local open source models for real applications was not really feasible yet. PrivateGPT proved otherwise, and the community responded.

That moment validated something important: the demand for private, local AI was not a niche preference. It was a fundamental need that the market had not yet addressed properly.

So we decided to go much further.

The two years of quiet work

After that initial explosion of interest, we made a deliberate choice. Rather than maintaining a popular but limited project in public, we forked our own repo and started building in private.

The honest reason was that there were too many unknowns to iterate responsibly in the open. What model architecture would hold up over time? What was the right tech stack? What indexing approach worked at scale? Agents or workflows? How do you manage long context without losing coherence? Tool calling was barely standardized, MCP did not exist yet, and harnesses as a concept were just emerging.

We needed to try, fail, and repeat, at a pace that would have been disruptive to an active open source community. So we separated the two tracks intentionally.

For two years, under the Zylon brand, we ran hundreds of iterations on the private AI stack. We did not just revisit PrivateGPT. We built and validated the full infrastructure layer required for companies across financial services, defense, healthcare, and government to actually adopt Private AI in production. Real deployments, real compliance requirements, real air-gap constraints, real enterprise workloads.

That work taught us things that are very hard to learn any other way.

Why now

The timing of this release is not arbitrary.

The AI ecosystem has matured and converged enough that several things are now true simultaneously. Model quality at the local level has improved dramatically. Standards like OpenAI-compatible APIs are well established. Tooling around structured output, function calling, and orchestration has stabilized.

On the demand side, something has shifted too. Data privacy and sovereignty have moved from a compliance checkbox to a strategic concern. Enterprise AI projects are stalling not because the models are too weak, but because legal, security, and infrastructure teams cannot sign off on sending sensitive data to external APIs. That is no longer a theoretical risk. It is a recurring obstacle in enterprise AI adoption, across sectors and geographies.

At the same time, token-based cloud costs have become genuinely difficult to justify at scale. What starts as an affordable pilot becomes a serious budget line when usage grows, and organizations have very limited control over how that pricing evolves.

Private AI addresses both issues directly. And PrivateGPT 1.0 is the most complete open source implementation of that vision we have ever shipped.

What PrivateGPT 1.0 actually is

Running a model locally is a first step. It is not enough.

To build useful AI applications you need a set of higher-level capabilities. A standard messages API. File and document ingestion. Retrieval with citations. Tool use and custom tool definitions. MCP connectors. Structured access to databases and CSV files. Web search and extraction. Code execution. Token counting, embeddings, and async workflows.

Until now, developers working with local inference had to build that layer themselves, or use a cloud AI API and accept the data sovereignty tradeoff. PrivateGPT 1.0 removes that choice.

The goal of the project is to bring a Claude-equivalent application API to your own infrastructure, so you can build private AI products without depending on cloud providers.

PrivateGPT 1.0 does not run models itself. It sits above any OpenAI-compatible inference server like Ollama, vLLM, llama.cpp, or LM Studio, and exposes a full application API on top. The architecture is deliberately simple:

Your app / agent / workflow / UI | PrivateGPT 1.0 API | Self-hosted LLM Server (Ollama, vLLM, etc.)

You choose the inference backend. PrivateGPT handles everything above it.

Where PrivateGPT fits in the stack

There are already excellent projects solving adjacent problems, and it is worth being precise about where PrivateGPT sits relative to them.

Ollama, vLLM, llama.cpp, LM Studio handle inference. They answer the question: how do I run a model? PrivateGPT does not replace them. It builds on them. Use both together: run your preferred inference server underneath, and use PrivateGPT as the application backend on top.

Onyx and Open WebUI are workspace applications. They are app-first experiences focused on chat and enterprise search. They are genuinely useful products. But PrivateGPT operates at a different level. It is not trying to be the final interface. It is the API layer underneath those kinds of products: the standardized local backend that handles messages, files, retrieval, tool use, data analysis, and orchestration. PrivateGPT ships with a lightweight UI for testing purposes, but the API is the product.

The clearest way to put it: inference below, apps above, PrivateGPT in the middle.

Full compatibility with the tools you already use

Because PrivateGPT implements the full Claude API spec, it works natively with any client or tool that integrates with Claude, including Anthropic's own first-party apps. That means Claude Code, Cowork, and the MS Office add-ins for Word, Excel, and PowerPoint can all run against a PrivateGPT instance, with all compute and data staying inside your own infrastructure.

Standard local inference servers cannot power those tools today, because they are missing key API capabilities such as structured output, tool use, and tokenizer endpoints. PrivateGPT covers all of that.

Beyond Claude-compatible tooling, PrivateGPT is also naturally compatible with the broader ecosystem of tools built around local inference providers: n8n, OpenCode, OpenClaw, Hermes, VSCode, Cline, and others.

What this means for Zylon

There is one more change that matters to us as much as the technical release.

Our commercial product, Zylon.ai, will now run on open source PrivateGPT under the hood. We have closed our private fork. From here on, we will be iterating PrivateGPT in public, and the work we do on the commercial side will flow back into the open source project.

That is not a marketing statement. It is a structural commitment. The community can see exactly what we are building, contribute to it, and hold us accountable to it. Zylon's commercial success is now directly tied to the health of the open source project. That alignment matters to us.

For the developers and organizations that have been following PrivateGPT since the early days, this is the project coming full circle: two years of hard-won learnings from real enterprise deployments, merged back into the community that made it possible in the first place.

What comes next

PrivateGPT 1.0 is the foundation. The roadmap from here includes deeper agentic capabilities, broader provider support, more built-in tools, and tighter integration with the deployment patterns we have validated across regulated industries through Zylon.

If you are a developer building AI applications on local infrastructure, PrivateGPT 1.0 gives you the backend you have been missing.

If you are an organization that needs enterprise-grade private AI with production support, compliance documentation, and the full managed stack, that is what Zylon is built for.

The open source project and the commercial product are now sharing the same core code, developed in the open, together.

We are glad to be back.