Private RAG Inside Your Own Microsoft 365 Tenancy
Retrieval augmented generation is a plain idea with an unfortunate name. Instead of asking a model what it remembers, you search your own documents first, hand the relevant passages to the model, and ask it to answer using only those. The answer comes back with citations you can open. Done properly it is the single most useful internal AI capability most Australian organisations can deploy, because it turns twenty years of accumulated documents into something a new starter can question in plain English.
Done carelessly it is an information disclosure incident with a friendly interface. The answer must respect who is allowed to see the source, and that is not a feature you switch on at the end. It is the design constraint the whole thing is built around. This page covers permission inheritance, why over broad file sharing is the first thing to fix, what indexing actually does to your content, and the cases where we tell people not to build it at all. To be clear about our own boundary: this runs inside the Microsoft 365 or Google tenancy you already own. We do not rack hardware, host GPUs or run model serving infrastructure.
Realistic ROI
Four Things That Decide Whether Private RAG Is Safe
The retrieval part is well understood engineering. These four are where projects succeed or quietly become a problem.
What retrieval actually does, in plain terms
When someone asks a question, the system searches your documents for passages likely to contain the answer, selects the best few, and passes them to the model along with an instruction to answer from those passages and cite them. The model is not being taught your content: your material never enters the model weights. It is reading a handful of excerpts you supplied, in that request, and writing an answer. The product around the model does keep records though, and it is worth separating the two. Prompts, responses and citations are stored as interaction history, and Microsoft Copilot now has a memory feature that persists user preferences across sessions in the user’s own mailbox. Both sit inside your tenancy and both have settings, which is why retention is a separate line item on this page rather than an afterthought. That distinction matters enormously in a security review, because it means the exposure question is about what the retrieval layer is allowed to fetch, not about what the model knows.
Permission inheritance is the whole game
The retrieval step must run as the person asking, so a user can only ever be given passages from documents they could already open. This is often called security trimming and it is what separates a legitimate internal search tool from an accidental disclosure. It has to work at retrieval time rather than being applied afterwards, because filtering an answer after the model has already read a restricted passage is not filtering at all. Test it deliberately with a low privilege account and a document that account should not see, and keep that test in your regression pack.
Over broad sharing is the number one thing to fix, before anything else
Almost every organisation we look at has libraries shared with everyone in the company by somebody solving a problem in a hurry, years ago. Salary bands, disciplinary letters, an acquisition model, a contract with unusual terms. Those files have always been exposed. They were protected by nobody knowing the filename. The moment retrieval is switched on, one plain English question surfaces them in seconds, and the incident report will say the AI project caused it even though the permission was set in 2019. Run a sharing audit and remediate before you index a single document.
Your content quality becomes visible immediately
Retrieval will happily quote the 2018 version of a policy, a draft somebody never deleted, and the current approved version, all in the same answer, with equal confidence. It cannot tell which one your organisation stands behind, because nothing in the file says. This is why scope and curation beat volume: a small set of current, owned, authoritative documents produces answers people trust, while indexing everything produces a system staff stop using after the second wrong answer. The unglamorous work of deciding what is authoritative is most of the value.
How a Private RAG Deployment Is Actually Built
Six stages. The first two decide whether the last four are worth doing.
Scope the corpus
Decide precisely which sites, libraries and folders are in scope, who owns each one and what the review cycle is. Resist the instinct to point it at everything, because the marginal document adds noise faster than it adds coverage. A good first corpus is one department’s current, approved material: policies, procedures, templates, the last two years of project documentation. Everything else can join later once the pattern is proven and the ownership question has an answer.
Audit and remediate sharing
Before any indexing, produce a report of what is shared with the whole organisation, what is shared with external guests, what has broken permission inheritance from its parent site, and what has no active owner. Then fix it. This is genuinely the hardest part of the project politically, because it means asking a lot of people whether a share from four years ago is still appropriate, and some of the answers will be embarrassing. It is also the part that protects everyone involved.
Index with permissions attached
Content is broken into passages and indexed with the access control information carried alongside, so that every search runs in the context of the requesting user. Where the platform’s native search already does this, use it rather than building a parallel index, because a second copy of your content is a second thing to secure, a second thing to keep current and a second place a deleted document can survive. Building a separate index should be a deliberate decision with a stated reason.
Answer with citations
Every answer names the documents it used and links to them, so a user can open the source and confirm. Configure it to say that it does not know when the retrieved passages do not contain the answer, which is harder than it sounds and worth insisting on. An internal tool that occasionally admits ignorance gets trusted. One that always produces something gets abandoned the first time somebody acts on a confident invention.
Freshness and deletion
When a document is updated the index must follow within a sensible window, and when a document is deleted or its permissions are tightened the index must reflect that promptly. Ask specifically how quickly a permission change propagates, because the gap between revoking access and the index honouring it is a real exposure window. Also decide what happens to conversation history that quotes a document later deleted under a retention policy.
Region, terms and audit
The whole thing runs inside your existing tenancy, pinned to an Australian region where the provider supports it for that specific capability, under your enterprise agreement with content excluded from foundation model training contractually rather than by reputation. Usage, queries and sources are captured in the tenancy audit log. Confirm the retention window on those logs early, because it is frequently shorter than the period an investigation would need.
What Private RAG Changes Day to Day
| Task | Traditional | Grounded in your own files | Notes |
|---|---|---|---|
| Finding the current policy | Search by filename and hope | Ask the question, get the clause and a link | Only works if somebody has decided which version is authoritative and archived the rest. |
| New starter questions | Interrupt the nearest senior person | Answered from the documented process | The most reliable value in the whole category, because the answers already exist in writing. |
| Tender and questionnaire responses | Copy from the last one and hope it is current | Drafted from approved source material with citations | A person still owns the submission. The tool removes the archaeology, not the judgement. |
| Contract and clause lookup | Someone remembers which agreement had it | Retrieved with the document named | Restrict the corpus tightly. Contracts are exactly the material that must be permission trimmed. |
| Technical and product knowledge | Scattered across old project folders | Searchable in plain English | Value depends entirely on whether anyone curated the folder or just never deleted anything. |
| A file the asker cannot open | Invisible unless they know it exists | Still invisible, by design | This is the control worth testing personally before you believe any vendor demonstration. |
| An answer that is wrong | Nobody notices for months | Traceable to the document that caused it | Citations turn a mysterious failure into a content fix, which is the whole reason to insist on them. |
| Adding a second department | A separate tool and a separate review | Same controls, new corpus | The governance work is done once, which is what makes departments two and three inexpensive. |
Where Private RAG Projects Go Wrong
Indexing before auditing internal sharing
This is the failure that turns a productivity project into an incident. Files shared with the entire organisation years ago become instantly discoverable by anyone who can phrase a question, and the material most likely to be in that state is HR, remuneration and legal correspondence, because those are the files people rushed. Run the sharing report first, remediate what should not be open, and get sign off from the site owners before indexing. If the project sponsor wants to skip this step to hit a date, that is the moment to push back in writing.
Confusing a demonstration with a permission test
Vendor demonstrations are run by an administrator who can see everything, which proves nothing about security trimming. Before you accept the build, take a real low privilege account, pick a document that account genuinely must not see, and ask several questions designed to extract its contents including indirect ones. Then repeat after revoking access to a document that was previously indexed, to measure how long the change takes to take effect. Those two tests belong in your acceptance criteria, not in a hopeful paragraph.
A second copy of your content nobody governs
Building a separate index creates another store of your material with its own retention, its own backups and its own access model, and it can outlive the source. A document deleted under a retention policy can survive inside an index, which is a genuine records management and legal hold problem. Prefer the platform’s native permission aware search where it does the job. Where a separate index is genuinely required, decide up front how deletion propagates, who administers it, and how it is included in your disposal schedule.
No owner for the content, only for the technology
The system is only as good as the documents behind it, and documents rot. Policies get superseded, procedures change, someone saves a draft next to the approved version. Without a named owner per content area and a review cycle, answer quality degrades steadily and staff stop trusting it, usually around three to six months in. Budget for content ownership as an ongoing role rather than a project task, and be honest with the sponsor that this is a permanent cost rather than a one off.
Scanned documents and images that hold the real answer
A surprising amount of important material is a photographed page, a scanned signed agreement or a diagram, and plain text extraction gets nothing useful from those. If your key content is in that state you either accept the gaps and say so clearly to users, or you fund text recognition and quality checking as a separate piece of work. What you must not do is let people assume coverage is complete when a category of documents is invisible to the system, because a confident answer built from partial evidence is worse than no answer.
Guest accounts and external sharing inside the corpus
Externally shared sites and guest users complicate the permission model considerably, because a guest may hold access granted for one project years ago that nobody has reviewed. Before indexing anything with external access, enumerate the guests, confirm each is still needed, and decide explicitly whether guests should have access to the retrieval interface at all. The default answer is usually no, and the safest scope for a first deployment is internal only content with no external sharing in it.
How Yes AI Approaches Private RAG
Sharing audit before anything is indexed
We produce the report of what is open to the whole organisation, what is shared externally, what has broken inheritance and what has no owner, and we work through remediation with the site owners. It is the least glamorous part of the engagement and the part that protects everybody.
A bounded first corpus, proven properly
One department, current approved material, a named content owner and an explicit test that a low privilege account cannot extract restricted content. We would rather deliver a small system people trust than a large one they stop using.
A clear line about what we do not do
We do not rack hardware, host GPUs or run model serving infrastructure. The deployment lives inside the tenancy you already own, in an Australian region, under your agreement. If somebody has told you private retrieval requires a server in your building, that is a much larger and usually unnecessary project.
Honest advice on when not to build it
If your documents are contradictory, undated and unowned, retrieval will surface that faster than it will help anyone, and we will tell you to fix the content first. We would rather scope a small documentation clean up than sell a system that quotes three versions of the same policy with equal confidence.
From Scattered Documents to a Tool People Trust
Five steps. Steps two and three take longer than anyone budgets for, and skipping them is how this goes wrong.
Pick the questions before the documents
We start from the twenty questions people actually ask each other every week, because those tell you which documents matter. Scoping from the file estate downwards produces an enormous corpus and a vague purpose. Scoping from real questions upwards produces something useful in weeks.
Audit and remediate sharing
A full report of over broad and external sharing across the sites in scope, remediation agreed with each site owner, and a named owner recorded for every library that survives into the corpus.
Curate what is authoritative
Identify the current approved version of each key document, archive superseded copies out of scope, and label the rest. This is a content governance exercise and the answer quality of the finished system depends on it more than on any technical choice.
Build, then attack it
Permission aware retrieval configured inside your tenancy with citations enforced, then deliberately tested with low privilege accounts, revoked access and indirect questions designed to extract content the tester should not see. Findings fixed before anyone else gets access.
Pilot, measure, then extend
One department for a few weeks with the queries reviewed weekly, wrong answers traced back to the document that caused them, content fixed, and only then a decision about the second corpus. Usage almost always reshapes the plan.
Related Reading
Private AI Deployment Australia
The full control set for AI inside a tenancy you already own.
AI Data Residency in Australia
Residency, sovereignty and sovereign AI told apart properly.
Sovereign AI for Australian Business
What procurement asks, and how to answer it honestly.
RAG Solutions Explained
The wider retrieval pattern and where it fits.
Microsoft Copilot (formerly Microsoft 365 Copilot)
What the built in option covers and where it stops.
SharePoint Premium
Content processing and governance on the platform you already run.
FAQ
Answers From Your Own Files, Without Breaking Access Control
Book a call. We start with the questions your staff actually ask, audit sharing before anything is indexed, and scope a first corpus worth trusting. Priced after scoping.
All discussions held in confidence. Australian-based consultants.