Using PrivateGPT with Microsoft Office, Locally

Start with PrivateGPT running locally

The first step is to have PrivateGPT running on your machine.

You can follow the documentation on PrivateGPT. The easiest way to get started is to run it on top of a local inference provider like Ollama.

Once PrivateGPT is running locally, it exposes a gateway that Claude-compatible applications can connect to. In my case, I have it running on:

localhost:8080

That local gateway is what makes the rest of the workflow possible.

Connect Microsoft Office to PrivateGPT

Open Microsoft Office and go to Add-ins.

From there, install the Claude add-in for Microsoft Office.

When the add-in opens, do not log in or sign up with a cloud Claude account. Instead, choose the Gateway option.

Then configure the gateway to point to your local PrivateGPT instance:

http://localhost:8080

If you have authentication enabled in PrivateGPT, you can also add your token there. If not, you can leave it simple for local testing.

After that, the add-in will ask for a model name. You can use any model name as the starter. For example, you can write claude, and PrivateGPT will route the request to the model you are running locally through your inference backend.

Once the connection says it is working, you are ready to use Microsoft Office with local AI.

An example: reviewing an NDA inside Word

For example, I can open an NDA in Microsoft Word and ask the assistant to review it.

A useful prompt could be:

“Find any potentially problematic clauses and add a comment explaining why each one may be risky.”

The Office add-in sends the request through the Claude-compatible gateway.

PrivateGPT receives the request locally.

The underlying model runs locally.

The document is analyzed locally.

And the comments are added directly inside Word.

That is the key point: you can use the same kind of workflow you would normally use with Claude, but without sending the document to a cloud AI provider.

Why not connect Office directly to Ollama?

A natural question is: why do I need PrivateGPT at all? Why not connect the Microsoft Office add-in directly to Ollama or another local inference server?

The reason is that tools like Ollama are inference servers. They are great at running models locally, but they do not implement the full Claude API.

Claude-compatible applications expect more than a simple text generation endpoint. They expect a full application API, including specific message formats, tool behavior, token handling, structured responses, and other capabilities.

That is the difference with PrivateGPT.

PrivateGPT implements the Claude API layer on top of your local inference backend. That is what allows Claude-compatible tools, including Microsoft 365 add-ins, Claude Code, and other applications in the Claude ecosystem, to work with local models.

In other words, Ollama runs the model.

PrivateGPT makes that model usable by Claude-compatible applications.

Why this matters

This is useful because it brings private AI directly into the tools people already use.

You can work inside Word, Excel, PowerPoint, coding tools, or other AI clients, while keeping the actual inference inside your own infrastructure.

You are not paying per token to an external model provider.

Your documents are not being sent to a cloud AI API.

And your AI stack remains under your control.

For companies working with sensitive information, this is the important part. Private AI is not only about running a model locally. It is about making that local model useful inside real workflows.

That is also the idea behind Zylon: giving organizations the infrastructure to run AI privately, with control over models, data, and deployment. And for teams that want to connect that private AI layer into internal tools, products, and automations, the Zylon API Gateway provides the governed integration layer.

PrivateGPT gives developers the open-source foundation for this.

Zylon brings the same private AI principle into enterprise production.

One important requirement: use a strong enough model

There is one practical thing to keep in mind.

These workflows can use a lot of context.

When a Microsoft Office add-in sends a document to the model, the request can easily involve tens of thousands of tokens. It may include the document, the user instruction, formatting context, comments, and other metadata.

So the model you use needs to be smart enough for the task, and it needs a large enough context window.

For something like reviewing an NDA, you want a model that can understand legal language, reason over long documents, and produce useful comments.

PrivateGPT gives you the compatibility layer.

The quality of the result still depends on the model you run underneath.

The bigger picture

The point of PrivateGPT is not just to run a chatbot locally.

The point is to make local AI compatible with the application ecosystem people already use.

That means you can use Claude-compatible tools, but route the work through your own infrastructure. You get the workflow experience of modern AI applications, with the privacy and control of local inference.

That is where private AI becomes really useful.

Not in a separate chat window.

Inside the work itself.

Author: Ivan Martinez Toro, Co-Founder & Co-CEO at Zylon
Published: June 8, 2026
Ivan leads private, on-premise AI deployments for regulated industries, helping financial institutions, healthcare organizations, and government entities implement secure, sovereign enterprise AI infrastructure.