Private RAG Inside Your Own Microsoft 365 Tenancy

Retrieval augmented generation means searching your own content for relevant passages and giving those passages to a model with an instruction to answer from them and cite them. The model is not trained on your material: it never enters the model weights, and it is reading excerpts you supplied for that one question. The product does retain records around that, which is a separate thing worth knowing. Prompts, responses and citations are kept as interaction history, and Microsoft Copilot has a memory feature that persists user preferences across sessions in the user’s own mailbox. Both live inside your tenancy and both are configurable, which is why retention is its own control rather than a footnote. This suits internal documents precisely because the alternative, fine tuning a model on your content, is expensive, hard to update, impossible to permission trim and produces answers you cannot trace back to a source. Retrieval keeps the source of truth in your document library where it belongs.

Done correctly, no. The retrieval step runs in the context of the person asking, so only documents that user could already open are eligible to be retrieved, and everything else is invisible to them. This is usually called security trimming and it has to happen at retrieval time rather than as a filter applied to the finished answer. The practical risk is not that the mechanism fails, it is that the underlying permissions were wrong to begin with. A file shared with the entire organisation is one a retrieval system will correctly serve to everyone, which is why the sharing audit comes before the build.

Because retrieval removes the accidental protection that filename obscurity has been providing for years. A remuneration spreadsheet in a team site that somebody opened to everyone in 2019 has been exposed the whole time, but finding it required knowing it existed and guessing its name. After retrieval is switched on, a single plain English question surfaces it in seconds. Nothing about the permission changed, only the discoverability, and the incident will nevertheless be attributed to the AI project. Fixing sharing first is both the right thing to do and the thing that protects the project sponsor.

On Microsoft 365 the deployment sits inside your existing tenancy, which can be pinned to an Australian region for the covered services including Copilot. On Google Workspace there is no Australian data region option as at August 2026, only the United States or Europe, so if onshore residency is a hard requirement then Workspace is the wrong platform for that particular workload. Either way, verify residency for the specific AI capability rather than assuming it inherits your tenancy setting, because coverage is published service by service and changes as products ship. On training, business and enterprise tiers of the mainstream products generally exclude customer content from foundation model training, but read the exact wording because it differs: Microsoft states flatly that prompts, responses and Microsoft Graph data are not used to train foundation models, while Google scopes its commitment to training outside your domain and outside Workspace and expresses it as applying without permission. Get the clause into your data processing terms rather than relying on a product page, and read the carve outs for abuse monitoring and preview features.

Often not, and the default should be to avoid one. The productivity platforms now provide permission aware search over your own content, which means the access control you already maintain is the access control the retrieval uses, with no second copy of your material to secure, keep current or delete from. A separate index is worth building when you need to combine content the platform cannot reach, such as an on premise line of business system, or when you need retrieval behaviour the native search cannot do. Where that is genuinely required, the index still runs inside your own cloud subscription under your agreement, not on infrastructure we operate. Treat it as a deliberate decision with a stated reason, and plan how deletion and permission changes propagate before you build it.

The technical build for a bounded first corpus is measured in days. The project is measured in weeks, and the variable is almost entirely how much sharing remediation and content curation is required. An organisation with well governed sites and a clear idea of which documents are authoritative can be live with a pilot group inside a month. An organisation with fifteen years of unreviewed sharing and three versions of every policy should expect the preparation to take longer than the build, and should regard that preparation as valuable in its own right rather than as project overhead.

When the documents are the problem. If your policies are undated, superseded copies sit beside current ones and nobody owns any of it, retrieval will quote all of them with equal confidence and staff will stop trusting the system within weeks. Fix the content first, even partially, and the same build becomes worth having. It is also the wrong project when the real requirement is a workflow rather than a question, for example approving something or updating a record, because retrieval answers questions and does not change anything. And if the useful knowledge lives in people’s heads rather than in files, there is nothing to retrieve and we will tell you so.