Do You Need a Custom LLM? A Plain-English Guide for Business Owners

Separate Model Choice From System Design

A custom model may mean fine-tuning an existing model or training a new one. A business can also use an existing model with instructions, document retrieval and tools. Those are different engineering decisions with different costs.

Fine-tuning can change patterns of behavior, such as a response format or task style. It is not a reliable substitute for looking up changing prices, availability or policy documents. A model can still produce an incorrect answer after training.

Define the problem first: what does the current system get wrong, how often does it matter, and how will you measure an improvement? That evidence is more useful than choosing a model approach from a label.

Test an Existing Model Against Representative Tasks

Build a small evaluation set containing normal requests, missing information, conflicting documents and tasks that should be escalated. Use answers reviewed by people who know the work.

Try clearer instructions, better source material and appropriate tools before assuming training is required. Keep a separate test set so improvements are not measured only on examples used during development.

There is no universal number of documents, words or monthly interactions that proves you need a custom model. Quality, response time, privacy requirements and total operating costs all matter.

What Retrieval Adds

Retrieval-augmented generation, or RAG, selects relevant material and supplies it to the model with the request. It can help the model answer from approved business documents without training those documents into the model.

It does not prevent every invented or incorrect answer. The system can retrieve the wrong document, miss a relevant passage or misinterpret the material. Test retrieval quality, source access and the handling of conflicting or missing information.

For a booking system, policy documents may explain the rules, while a live calendar tool supplies availability and creates the appointment. Retrieving a document alone does not provide a live booking integration.

When Fine-Tuning or Private Hosting May Be Worth Testing

Fine-tuning may be worth evaluating when a recurring behavior remains difficult to achieve with instructions and you have suitable, authorised training examples. Compare it with the existing approach on the same held-out tasks.

Private hosting is a separate choice. An existing model can be privately hosted without training a new model. It may help meet a deployment constraint, but responsibility for access controls, updates, monitoring and infrastructure remains.

Neither fine-tuning nor private hosting guarantees legal compliance, confidentiality or error-free output. Verify the full data flow and the controls required for your use case.

Compare the Full Cost

Request costs for data preparation, evaluation, engineering, model usage or compute, hosting, monitoring and ongoing maintenance. Include the people who review failures and update the source material.

A useful comparison uses your request volume and required response time. A lower price per model call can be outweighed by infrastructure or support costs. Conversely, a well-tested change may reduce the number of attempts needed to finish a task.

Treat supplier estimates as estimates until a trial establishes actual usage and review work. There is no fixed percentage of benefit or cost saving that applies to every business.

Make the Decision From the Trial

Write down the task, the baseline result, the proposed change and its cost. Identify the remaining errors and whether an operational control can manage them. If the simpler system meets the requirement, further model work may have little value.

Book a 30-minute consultation to discuss your requirements. It is free for businesses with 20 or more full-time staff; otherwise AUD $200 including GST.