A maintenance method for internal knowledge systems

Find stale answers by tracing changes from the source to the response

A knowledge base can contain the current document and still give staff an old answer. The source may have been updated while an index, saved summary or generated response retained the previous rule. Auditing freshness means checking the whole path from an approved change to the answer a real user receives, under that user’s permissions.

This guide describes a proposed audit for internal business knowledge. It separates source age from actual change propagation and uses hypothetical examples. A quiet document library is not automatically stale, and a recent indexing timestamp is not proof that the right content reached the answer.

Scope a stale answer audit (03) 9003 0111

Knowledge freshness audit tracing an approved source change through ingestion, retrieval and the staff-facing answer. Select the diagram to view it full size.

Four things a freshness audit must establish

Authority

Which version should govern?

Identify the approved source, its owner and any effective-date conditions.

Change

What actually changed in the source?

Compare content and status rather than treating a timestamp as the whole story.

Propagation

Where did the change reach?

Inspect ingestion, search results and downstream summaries separately.

Answer

What does the intended user receive?

Test the final response with representative questions and permissions.

Freshness is a relationship, not an age label

A document can remain correct for a long time. A newly processed document can still be the wrong version. Audit the relationship between source authority, content and answer.

Distinguish no change from missed change

A sync process may have nothing to write when approved sources have not changed. That is different from failing to ingest a real update. Establish whether the source changed and whether the downstream content matches it. Alerting purely on time since the last write can create noise while failing to identify the actual stale-answer pathway.

Include derived content in the inventory

Answers can depend on extracted text, search fragments, summaries, saved templates and cached responses. Updating the original document does not necessarily update every derivative. Map which artefacts feed the staff-facing answer and keep a relationship to the source version. Without that map, a team can repeatedly refresh the library while the stale summary remains untouched.

Test deletion and withdrawal

A policy owner may withdraw a document rather than publish a replacement. The audit should check whether removed or unapproved material remains retrievable or influences answers. Preserve historical records under the organisation's policy, but distinguish historical access from current authority. A search result should not silently promote a withdrawn procedure because its wording matches the question well.

Use known changes as control cases

Select an approved change whose old and new meanings are clear, then trace it through the system. Include a source that has not changed as a comparison. This tests whether the audit can identify both healthy quiet behaviour and a real missed update. A report of recent timestamps alone cannot establish either result.

A six-part stale answer audit

Work from approved sources towards the user-facing answer. Keep read failures and missing evidence separate from confirmed stale content.

A source authority map

Inventory authoritative sources

List the source collections, owners, status rules and the questions each collection is meant to answer. Identify copies that staff may mistake for current policy. Record how approval, withdrawal and future effective dates are represented. Start with a bounded question group rather than scanning every file without knowing which should govern. The inventory provides the reference against which freshness can be assessed.

Known changed and unchanged cases

Select meaningful control changes

Choose approved examples where a material instruction changed, such as an escalation owner or required form. Retain the previous and current content under authorised access. Add unchanged controls and withdrawn documents. Record the expected answer for each relevant question. These cases make the audit capable of detecting specific propagation failures instead of merely listing files by age.

Source-to-ingestion evidence

Inspect ingestion evidence

Check whether the ingestion process saw the correct source version and extracted the relevant content. Compare the actual text or a suitable content fingerprint, not only job completion time. Record unreadable files, access failures and unsupported formats. A job that completed after skipping the changed source should not count as successful propagation for that source.

A map of stale downstream content

Check retrieval and derivatives

Run the relevant questions against the search layer and inspect the passages returned. Then identify summaries, templates or caches that may still carry the old instruction. Keep the source and version relationship visible. If retrieval returns both old and current material, assess whether authority metadata is used correctly rather than assuming the answer generator will infer which one applies.

User-level outcome evidence

Test real answers and access

Ask representative questions as users with the permissions involved in the workflow. Check the actual response, references and conditions. Include questions that previously triggered the old answer and variants that may retrieve a different passage. Verify that restricted sources do not leak through retained summaries and that a broken reference is reported as unavailable evidence.

A verified propagation repair

Repair the cause and prove recovery

Correct the specific source, ingestion, retrieval or derivative issue under the agreed scope. Re-run the failed control and an unchanged control after the repair. Confirm the final answer now uses the approved version and that old content is handled according to policy. Document the cause and the detection method so the same pathway does not remain invisible.

Freshness scenarios that need different verdicts

Task	Traditional	Audit interpretation	Notes
Approved document has not changed	Old modification date looks unhealthy	Compare downstream content with the approved source	A quiet source can be current; the audit should not invent work solely from age.
Document changed but sync wrote nothing	Job status says completed	Check whether the changed source was seen and extracted	The absence of a downstream update matters because there is a known source change.
Index updated but summary stayed old	Search timestamp looks recent	Inspect the derivative used by the answer	The stale instruction may survive outside the main search index.
Policy withdrawn without replacement	Old text still answers queries	Check current-authority filtering and withdrawal handling	Historical retention should not make withdrawn guidance appear operative.
Future policy uploaded early	Newest file wins	Apply approved effective-date and status rules	Current guidance and upcoming changes may both need clear labels.
Different roles see different answers	Administrator test passes	Test representative user permissions	Access failures and cached content can change what an ordinary employee receives.
Reference link breaks after a move	Answer text remains plausible	Mark the evidence unavailable and trace the source identity	A changed URL should not silently detach an answer from its authoritative document.
A copy in another folder is older	Both files look relevant	Identify the governing source and copied derivative	Fixing only the main library may leave a competing old answer path.

Freshness checks that create false confidence

A recent job is treated as current content

A run timestamp proves that a process ran, not that it read the changed file or updated the answer. Inspect source identity and content at the relevant stages. Report skipped or failed reads as evidence gaps rather than successful refreshes.

Every old document is called stale

Some procedures remain valid without frequent edits. Ask the owner about authority and compare the downstream copy with the approved source. Use review dates for governance where appropriate, but do not confuse an overdue content review with a demonstrated propagation failure.

Only the main index is checked

Generated summaries, templates and saved answers can outlive the source content that created them. Include those derivatives in the map and test their invalidation or refresh process. A correct search result does not guarantee that the final response uses it.

The audit changes live policy to create a test

Use approved control changes or a safe test collection. Do not insert artificial instructions into a real policy library merely to prove that indexing works. The audit should not create a new business risk or leave a sentinel that staff may later retrieve as genuine guidance.

A failed read becomes a clean result

If the auditor cannot access a source or inspect a derivative, the status is unverified. Keep that limitation visible and resolve the access or evidence gap. Zero observed stale records means little when the important collection was not actually examined.

The repair addresses only one question wording

A cached answer may be fixed for one phrase while another path still retrieves the old text. Test relevant variants and inspect source relationships. Report the coverage honestly rather than declaring an entire knowledge base current from a single successful response.

Worked example: the escalation owner changed, but the answer did not

Imagine a hypothetical operations procedure changes the owner for a particular escalation. The approved document is updated and the document portal displays the new version. Staff still receive the old owner when they ask the internal assistant whom to contact. An audit that checks only the portal or the latest sync timestamp will miss the problem because both appear current.

Start by confirming the approved change and the exact question context. Preserve the old and new passages under authorised access. Check whether ingestion extracted the current paragraph. If it did, inspect the retrieval result for the staff question. If retrieval also returns the new owner, look at the summary or template supplied to the answer. The old instruction may persist in a derivative that was generated earlier and never refreshed.

The repair should address that derivative relationship rather than editing the final answer by hand. Establish how a source change identifies affected summaries and how those summaries are refreshed or withheld until checked. Re-run the original question and relevant variants. Inspect the references and test under an ordinary staff role, because an administrator may have access to a source that the employee cannot retrieve.

Include an unchanged procedure as a control. Its content should remain available and correct without requiring an artificial edit. This helps show that the freshness check measures whether downstream guidance matches the approved source, rather than demanding activity for its own sake. The final report should identify the specific stale stage, the repair and the questions tested, without claiming that every answer in the organisation has been audited.

A change checklist for document owners

When approving a material change, record the source identity, status, effective conditions and a short description of the decision that changed. Identify the staff questions likely to be affected. If the document is replacing another source, record that relationship explicitly. This gives the knowledge system a clearer maintenance signal than a renamed file or an unexplained upload timestamp.

After propagation, test an answer that should change and one that should remain the same. Inspect the supporting passage and confirm the reference opens for the intended role. Check relevant summaries, templates and saved responses that depend on the source. If a stage cannot be examined, record it as an evidence gap rather than treating a completed sync as proof that all derivatives are current.

Assign a person to investigate failed propagation and another business owner to settle content conflicts where appropriate. Keep the technical repair distinct from policy approval. The system may be faithfully repeating a document that the business has not formally superseded. In that case, changing the model is not the right remedy; the source authority and approval process need clarification first.

Separate content review, ingestion health and answer freshness

A policy owner may need to review a document even when the knowledge system reproduces it perfectly. That is a content-governance issue. An ingestion job may fail while the unchanged approved content remains available. That is an operational issue. A final answer may use a superseded instruction despite healthy ingestion. That is an answer-freshness issue. Keep these findings separate so each reaches the person who can actually resolve it.

For content governance, record the owner, approval status and any review requirement established by the business. For ingestion health, record the source coverage, failed reads and processing evidence. For answer freshness, compare the user-facing response with the approved source and relevant conditions. A combined dashboard can show all three, but it should not collapse them into one green badge that hides which relationship was tested.

Consider a quiet collection where no approved source has changed during the observation period. A last-write alert may report a problem even though every answer control remains current. Investigate whether the process can still detect changes, using a safe control or existing approved update. Do not force a live policy edit just to satisfy a timestamp rule. Monitoring should demonstrate readiness and propagation where relevant, without creating meaningless content churn.

Consider the opposite case: a scheduled job completes successfully after losing access to a restricted collection. The report may look healthy while important sources were skipped. Record coverage explicitly and test with a question that requires the affected material. The correct result is an evidence or access gap until the source can be examined. This distinction helps the business avoid both noisy false alarms and broad health claims based on incomplete observation.

How Yes AI can help trace outdated guidance

Map the answer path

We can scope the collections and derivatives involved in one recurring question group. The map should identify source owners and the points where content is copied or transformed. This creates a practical route for diagnosing why a current file still leads to an old response.

Build a meaningful audit set

We can help select approved changed, unchanged and withdrawn examples, with expected answer requirements. A proposed audit can then prove whether it detects the relevant states. This is more useful than a report that simply labels old timestamps as unhealthy.

Investigate propagation failures

A scoped review can compare source content, ingestion evidence, retrieval results and final answers. We can separate access problems from stale content and document the specific repair required. Provider interfaces and available logs need confirmation before promising a particular diagnostic depth.

Keep maintenance proportionate

If staff rely on a small set of stable procedures, clear ownership and periodic manual answer checks may be sufficient. A complex monitoring layer is not automatically justified. We can recommend a simpler review process when it addresses the real exposure without adding unnecessary infrastructure.

Audit one knowledge area from source to answer

Choose a named content owner and a bounded set of questions. Keep any production changes separate from the read-only audit findings until their scope is agreed.

Establish authority

Identify approved sources, status rules and derivative content. Record the questions the collection should answer and the roles that need access.

Choose control cases

Select known changed, unchanged and withdrawn examples under approved access. Define the expected current answer and the evidence that demonstrates each state.

Trace propagation

Compare actual content through ingestion, retrieval and downstream summaries. Preserve failed reads and missing logs as limitations rather than assuming those stages are healthy.

Check user outcomes

Ask relevant questions under representative permissions. Inspect answers, source versions and reference access, including variants that may expose another retrieval path.

Repair and maintain

Address the demonstrated cause within the agreed scope, re-run failed and unchanged controls and assign ownership for future content changes and review.

Turn this guide into your next steps

Use these steps to prepare your own review. Tick a step once you have recorded its evidence. Ticks are temporary and are not saved or sent to us.

Bring one example of the process you want to improve. We can help define the scope, checks and next decision. Consultation options and any fee are shown before you book.

Scope a stale answer audit

FAQ

How often should a business knowledge base be refreshed?

The right cadence depends on how sources change and how quickly staff need those changes reflected. Establish the required relationship between an approved change and the answer, then verify the process meets it. A fixed refresh schedule alone does not prove correctness, and a quiet source does not necessarily need repeated writes.

Does an old last-indexed date mean answers are stale?

Not by itself. Compare the downstream content with the approved source and establish whether the source changed. If nothing changed, the content may still be current. If a known update was missed, investigate propagation. Keep content-review governance separate from evidence that the system is behind its source.

Why can the right document produce an old answer?

The answer may use an older search fragment, saved summary, template or cached response. It may also retrieve a competing copy with unclear authority. Trace the exact evidence supplied to the answer and its source version. Updating the original file is only one part of maintaining the complete answer path.

Should old documents be deleted?

That depends on the organisation's retention and records policies. The audit should distinguish retaining history from treating it as current guidance. Withdrawn or superseded material may remain accessible to appropriate users while being excluded from ordinary current-policy answers. Do not make deletion an automatic side effect of a freshness finding.

Can we test freshness by inserting a fake instruction?

Use a safe test collection or an approved real change instead. Adding artificial content to a live policy source can create a misleading answer for staff and may persist in derivatives. A controlled fixture should be clearly separated from business guidance and removed according to the test plan after its purpose is complete.

What should a stale-answer report contain?

Include the question, intended user role, authoritative source version, observed answer, stale passage and the stage where propagation diverged. State evidence gaps and the tested coverage. Recommend the specific repair and a repeatable check. Avoid a broad healthy verdict when only timestamps or one administrator session were inspected.

When is a manual audit enough?

A bounded library with stable content and a clear owner may be adequately served by a small set of recurring answer checks and a change checklist. More automation may be justified when sources change frequently or there are many derivatives and users. Choose the maintenance method around the observed failure routes, not the size of the technology stack.

Find where the old answer survives

Bring a question that still produces outdated guidance and the approved document that should answer it. Yes AI can scope a trace from source change to the response staff actually receive.