Achieving a best practice approach to Large Language Model (LLM) solutions

However, while convenience offers a compelling reason to use LLMs, such new territory comes with new risks. How can we be sure of the accuracy and quality of information, particularly when using LLMs in a professional capacityt? Is it a reliable tool, and what are the key considerations when seeking a solution?

Consider information sources

When exploring different LLM solutions, taking time at the initial stages to ensure any concerns on accuracy and reliability are answered is essential. There are currently three primary sources of information used to drive LLMs to consider. First, some models use a training data set as its prime source of knowledge. Second is information gained by directly connecting to the internet to find answers, sometimes providing a link to the source for further research. Finally, there is the ability to use a company's proprietary information and documents to populate the LLM's knowledge base.

In terms of accuracy, using a training data set from a third-party source is the least reliable, as the origin of the information gathered is unknown. In comparison, LLMs that use company documents offer the highest level of reliability, which is particularly important to foster knowledge reuse and efficient information access.

Understand the human-machine interaction

While LLMs can prepare the groundwork and input information into a report in seconds rather than days, texts produced can often be general and superficial in content. Therefore, it’s essential to treat LLMs as co-pilots when writing memos and reports or summarising long documents. A human quality check is still necessary to validate and polish the final results.

This may result in many iterations before you are happy that the final result is customised to the end user's needs. Keeping quality top of mind helps to ensure you are not producing information that lacks rigour and leads to more questions from the end user. While this approach may slow the process down, using human knowledge and expertise is an important final step in keeping the quality of information high.

Build model-appropriate training

Adopting an LLM solution should always include model-specific training. For example, the training should emphasise the model's limitations and the risks of over-relying on its capabilities. A good training programme also provides the chance to underline the importance of data security and confidentiality.

Efficient use should also be taught. This is about knowing how to prompt and interact with LLMs to get a better answer than you would otherwise. Rather than simply providing training on how to use the model, clearly demonstrating how use improves the quality of tasks provides a more credible foundation for adoption.

Next steps

It’s important to remember that it’s called artificial intelligence for a reason; there is no real intelligence behind it. The model simply tries to guess the next word and sentences that fit the requests prompted by the user. The quality of those words and sentences will also depend on the information source the model uses or has been trained to use. If companies want to maintain quality of work, human oversight remains an essential part of the process.

Of course, this level of human involvement may vary and evolve depending on the generative AI (genAI) use case and should be evaluated taking into consideration the risk profile of the genAI technology and the task it is running. If we take the audit function as an example, the use of LLMs requires transparency between the client and external auditor to ensure quality and confidentiality are aligned and maintained. During the field work, It’s important that auditors have the appropriate knowledge and skills to identify, evaluate and respond to risks of material misstatement related to the use of genAI.

Finally, six months is a long time in any AI landscape. The speed of technological change means solutions that aren't possible now could very well be a few months down the line. This dynamic environment presents a opportunity for dedicated technology teams to stay ahead of the curve and harness the full potential of LLMs. Companies that do not have the size or scale to do this can look at partnerships and external experts to guide them, inspiring them to keep learning and adapting to achieve best practice when it comes to LLM solutions.