The problems: LLM hallucinations and limited context
LLMs often take a long time using expensive resources to train, sometimes months of run time using dozens of state-of-the-art server GPUs such as NVIDIA H100s. Keeping the LLMs completely up-to-date by retraining from scratch is a non-starter, although the less-expensive process of fine-tuning the base model on newer data can help.
Fine-tuning sometimes has its drawbacks, however, as it can reduce functionality present in the base model (such as general-purpose queries handled well in Llama) when adding new functionality by fine-tuning (such as code generation added to Code Llama).
What happens if you ask an LLM that was trained on data that ended in 2022 about something that occurred in 2023? Two possibilities: It will either realize it doesn’t know, or it won’t. If the former, it will typically tell you about its training data, e.g. “As of my last update in January 2022, I had information on….” If the latter, it will try to give you an answer based on older, similar but irrelevant data, or it might outright make stuff up (hallucinate).
To avoid triggering LLM hallucinations, it sometimes helps to mention the date of an event or a relevant web URL in your prompt. You can also supply a relevant document, but providing long documents (whether by supplying the text or the URL) works only until the LLM’s context limit is reached, and then it stops reading. By the way, the context limits differ among models: two Claude models offer a 100K token context window, which works out to about 75,000 words, which is much higher than most other LLMs.
The solution: Ground the LLM with facts
As you can guess from the title and beginning of this article, one answer to both of these problems is retrieval-augmented generation. At a high level, RAG works by combining an internet or document search with a language model, in ways that get around the issues you would encounter by trying to do the two steps manually, for example the problem of having the output from the search exceed the language model’s context limit.
The first step in RAG is to use the query for an internet or document or database search, and vectorize the source information into a dense high-dimensional form, typically by generating an embedding vector and storing it in a vector database. This is the retrieval phase.
Then you can vectorize the query itself and use FAISS or another similarity search, typically using a cosine metric for similarity, against the vector database, and use that to extract the most relevant portions (or top K items) of the source information and present them to the LLM along with the query text. This is the augmentation phase.
Finally, the LLM, referred to in the original Facebook AI paper as a seq2seq model, generates an answer. This is the generation phase.