❯ Guillaume Laforge

Some advice and good practices when integrating an LLM in your application

large-language-models machine-learning best-practices patterns

When integrating an LLM into your applicaton to extend it and make it smarter, it’s important to be aware of the pitfalls and best practices you need to follow to avoid some common problems and integrate them successfully. This article will guide you through some key best practices that I’ve come across.

Understanding the Challenges of Implementing LLMs in Real-World Applications

One of the first challenges is that LLMs are constantly being improved. This means that the model you start using could change under the hood, and suddenly your application doesn’t work as it did before. Your prompts might need adjustments to work with the newer version, or worse, they might even lead to unintended results!

Furthermore, you need to consider how to effectively manage your prompts, especially when your applications start to get more complex. Prompts can easily become convoluted and difficult to modify. Imagine having to comb through a hundred lines of code in your application to find a specific prompt just to tweak a single word. That’s why prompt externalization (not keeping your prompts inside your source files) is going to be important, to easily find your prompts, and have a birds-eye view of all of them!

Keeping track of changes and ensuring that the LLM’s behavior remains consistent throughout your development process is another challenge. How can you ensure that a particular feature still functions correctly after upgrading your prompts, or even changing model versions? You need to version your prompts (we’ll cover that in more details in a moment). Think of your prompts like code — just like your software code, prompts should have version control for easy management. Versioning ensures that you can quickly revert to previous versions if necessary, while providing a helpful audit trail to see exactly what changes have occurred in your prompt management process.

Prompt Engineering for Consistent and Effective LLM Applications

You’ve probably noticed that one of the main things that determines how well your application works with a Large Language Model (LLM) is the prompt you use to guide it. Prompts act like a guidebook for the LLM, explaining what you expect from it and how it should format its response. You’ve likely heard about the importance of using good prompts, but how do you go about creating prompts that are reliable and adaptable in the long run?

Think of your prompts like code artifacts. Just as you version your code to keep track of changes and ensure consistency, you should also version your prompts. This allows you to:

  • Keep track of your prompt evolution: You’ll have a clear record of how your prompts have changed over time, which helps you understand the application’s evolving behavior.
  • Create a helpful audit trail: Having versions of your prompts will help you understand exactly how the application behaved at specific times. This is essential for debugging, diagnosing issues, or understanding how user feedback impacted your application.
  • Enable rollbacks: If you encounter an issue or want to test different prompt versions, you can easily revert to a previous state to ensure that you can isolate problems, revert to previously working versions, or simply experiment with different phrasing.

But simply versioning prompts isn’t enough. Imagine you need to make a change to one particular prompt in a massive LLM-powered application. It might involve a lot of tedious code-hunting. That’s where prompt externalization, that we mentioned earlier, comes in! Externalizing prompts is all about taking them out of your code and treating them like a separate configuration file. This way, they are:

  • Easy to modify: Changing your prompts becomes a breeze. Just go to your external prompt file, make the adjustments, and you’re ready to go! No more scouring through complex code to find a single prompt in some string variables somewhere.
  • More flexible: By using externalized prompts, you can easily experiment with different versions or phrasing without rewriting your entire application. This lets you quickly adapt your prompts in response to user feedback or changes in your model.
  • Easier to manage: Keeping prompts in their own dedicated file makes it easy to maintain them, making sure that your prompts are consistent and up-to-date. This approach becomes increasingly valuable as your applications become more complex and you have a growing set of prompts to maintain.

There are open source projects, or open formats that have emerged recently, to externalize prompts. For examples Firebase’s GenKit LLM framework came up with their dotPrompt format, which not only externalizes the prompt itself, but also the name of the model, its configuration (temperature, etc.)

Model Versioning: Preventing Surprises

Now let’s cover the importance of managing model versions, a critical aspect of ensuring that your LLM-powered application continues to work reliably.

Imagine that you’ve built a great application using a particular model, and you’re proud of the results. However, what happens when the LLM provider releases an updated version? It might offer performance improvements, but the updates can also change how the model responds to your prompts, potentially leading to unexpected issues or even breaking your application.

To avoid these unexpected changes, the key principle is to pin the specific version of the LLM model that you use for your application. For example, when using Gemini 1.5 Pro, if you use the version gemini-1.5-pro, you’re actually using the latest version of the model. Currently, it’s gemini-1.5-pro-001. But if tomorrow Google releases gemini-1.5-pro-002, your application would suddenly start using that new version. So be very explicit in the model version.

Here’s why this is essential:

  • Avoid Drifting Model Behavior: The update to an LLM might come with subtle changes that can lead to a shift in the model’s responses, and you may not always be able to anticipate these changes beforehand. This can lead to inconsistency, where a prompt that generated a certain output in one version of the model generates a completely different (and perhaps undesirable) output in a newer version.
  • Maintain Application Consistency: To keep your application performing reliably, you want to control the LLM’s responses as much as possible, and pinning the model version ensures that you can do this. If you’re using a specific model, the prompts that are part of your application work in the context of that model’s specific training and behaviors. Pinning the version helps you avoid unexpected changes that may interfere with your prompts’ effectiveness.
  • Simplify Auditing and Debugging: In case of an unexpected issue or an unexplained change in your LLM’s behavior, being able to easily trace back the specific model version that’s running provides invaluable context for debugging and understanding why those changes occurred. It helps isolate issues to specific model versions, so you can resolve them quicker.

While using the latest and greatest LLM version might seem tempting for its improved capabilities, remember: the consistent performance and reliability of your application should be a top priority. By pinpointing the model version you use, you gain better control over its behavior and maintain a smooth and predictable experience for your users.

Optimizing for Efficiency: The Power of Response Caching

Even with well-crafted prompts, pinned versions, generating responses from a Large Language Model (LLM) can still be expensive. This is where response caching comes in, offering a crucial way to improve both the performance and the cost-efficiency of your application.

Models like Gemini support context caching. Quoting the documentation:

Use context caching to reduce the cost of requests that contain repeat content with high input token counts. Cached context items, such as a large amount of text, an audio file, or a video file, can be used in prompt requests to the Gemini API to generate output. Requests that use the same cache in the prompt also include text unique to each prompt. For example, each prompt request that composes a chat conversation might include the same context cache that references a video along with unique text that comprises each turn in the chat. The minimum size of a context cache is 32,768 tokens.

By caching frequently used responses or heavy multimodal documents, you avoid having to generate them over and over again, leading to a dramatic improvement in performance and a reduction in LLM usage costs. Imagine users frequently asking the same question, like “What are the benefits of using your app?". By caching the response to this question, you’ll be able to provide users with a fast and efficient response without burdening the LLM each time.

But how do you actually implement caching? You can choose different strategies for your caching system, each with its own benefits:

  • Context Caching: If your model, like Gemini, supports caching already, be sure to understand how it works, what can be cached or not, the pros and cons, or potential limitations.
  • Basic Caching: Store LLM responses based on the exact input. If you encounter a query that you’ve already generated, you can provide a pre-cached response, saving on processing time. You could also do some minimal string modifications to normalize whitespace, put everything in lowercase, etc, to get the chance to cache very similar prompts.
  • Advanced Caching with Similarity Search: Even with string normalization, you might find that users don’t always ask the exact same question, but the query can still be extremely similar. Think of typos, minor word substitutions, synonyms, or variations in punctuation. Instead of treating every query as unique, consider approximate nearest neighbor search and embedding vector similarity. This approach helps you find queries that are nearly identical, even with minor variations. You can then leverage this functionality to serve the same cached response for queries that are semantically similar, increasing the effectiveness of your caching strategy and ensuring that you only compute distinct queries once. For vector similarity, make sure to test with different inputs, to find the right threshold to say that a new prompt is equivalent to an older cached prompt/response pair.

Caching responses not only speeds up your LLM-powered application, lowering the perceived latency, but also significantly cuts down on LLM usage costs, helping you keep your application running smoothly while maximizing cost-effectiveness.

Building Safeguards: Ensuring Robustness with Guardrails

Let’s shift our focus to building safety mechanisms. This is crucial for creating reliable, trustworthy applications. Enter the concept of guardrails, which are safety systems designed to protect your application and users from unexpected, unwanted, or even harmful outcomes.

Think of guardrails like a protective fence, ensuring that the LLM stays within safe boundaries while performing its tasks. Imagine if someone tried to make an inappropriate request, or worse, a request that could cause harm. This is where guardrails step in.

Guardrails serve two main purposes:

  • Input Validation: Guardrails can examine the user input and determine whether it’s acceptable for your application and whether it aligns with your intended use case. Imagine preventing your LLM from processing prompts with malicious language or data that could cause harm to users.
  • Output Filtering: Guardrails are not only for examining the user’s input but also for checking the outputs of the LLM. By analyzing the LLM’s generated responses, you can filter out inappropriate content or responses that don’t meet your requirements.

What are the three primary types of guardrails?

  • Model’s safety settings: Models have usually been fine tuned to avoid certain harmful content in both input and output. They also give you access to safety settings, with different harm categories and safety thresholds. You should test those settings and how they can be configured for your use case. For example, have a look at the available safety filters for Gemini.
  • Static Guardrails: These are predefined rules that are set before the LLM begins to process any input. These can be rules that dictate certain formatting for prompts, limitations on input length, or even basic checks for prohibited terms or requests. Static guardrails offer fast processing times, since the checks are performed beforehand on the input strings, in your own code.
  • Dynamic Guardrails: These are flexible guardrails that work in conjunction with the LLM or the embedding model used for text classification, continuously adapting to changes in user input or the output of the model itself. They allow you to handle complex or unpredictable situations and perform nuanced checks to maintain the safety and integrity of your application. You might have a look at Google Cloud Natural Language Processing’s moderation endpoint, or the free Perspective API used by newspapers.

When you implement guardrails, it’s also critical to consider performance impact. You want to make sure that these safeguards don’t add too much latency and delay user experience. That’s where parallel processing can come into play! Instead of waiting for the guardrail check to finish before starting the LLM generation, consider launching both tasks in parallel, optimizing speed and efficiency without sacrificing safety. Then, if the guardrails raise a red flag, you can stop the response generation, and reply right away to the user that the input content was problematic. For the response, unless you have a streaming kind of guardrail system, you might have to wait for the whole response to be generated before evaluating it with the guardrail, in which case, you can’t really do parallel processing.

Always remember: Guardrails should be continuously refined and updated as you identify new potential risks. Gathering feedback from users, giving them the ability to report a bad response is one approach. But you should also monitor your application LLM responses to do some vibe-checks at random to ensure your application is behaving correctly.

Evaluating and Monitoring for Consistent Performance

Onto the most crucial aspects of any application, regardless of its technology, is evaluation and monitoring. This is essential for ensuring your LLM-powered application continues to function reliably and meets your expectations as it interacts with users in the real world.

Imagine you make an update to your application, or perhaps even a simple tweak to one of your prompts. Without proper monitoring, you won’t know if those changes had unintended consequences. You could end up with an app that gives unexpected results, leads to user frustration, or even creates unforeseen safety issues. That’s where a robust evaluation and monitoring framework comes into play!

Your LLM-powered app needs a systematic way to ensure that everything is running smoothly and effectively. You need to:

  • Establish Evaluation Metrics: You need clear guidelines to judge the LLM’s performance. Think of key metrics like accuracy, relevance, and coherence.

    • Accuracy: This measures how often the LLM generates correct and factually accurate responses. This is particularly crucial if your application is designed for providing reliable information or carrying out fact-based tasks.
    • Relevance: You need to make sure the LLM stays focused on the core issue. It should respond to your prompts in a meaningful and helpful way, instead of giving irrelevant or off-topic responses.
    • Coherence: You need to check if the LLM produces well-written and logical text. Coherent responses are easily understood by users and don’t leave them feeling confused or disoriented.

  • Gather User Feedback: It’s essential to go beyond just numbers. Your application’s performance shouldn’t just be evaluated on your own terms. Get feedback from the users, gather data on how they are using the application, and check their satisfaction with the outputs of your application. You can even ask users to provide their opinions on specific generated answers, giving you valuable insights into what resonates with them and how you can improve. Consider using tools like “thumbs up” or “thumbs down” buttons, offering an easy way for users to indicate their sentiment towards the LLM’s responses, or a way to report and explain what wasn’t up to the level of their expectations.

  • Build a “Golden Responses” Dataset: Create a collection of carefully chosen inputs and their desired, accurate responses. These “golden” examples act like benchmarks, helping you measure how closely the LLM matches your expected results for specific tasks. By periodically checking how your LLM performs against these golden examples, you can get a clear picture of potential issues and make necessary adjustments. You can use this set as a starting point to track potential regressions and make sure the LLM’s behavior is aligned with your expectations.

  • Implement Continuous Monitoring: Monitoring shouldn’t be a one-time event. It’s an ongoing process, like keeping a watchful eye on your application as it functions in the real world. By monitoring in real-time, you can detect anomalies, unexpected issues, or performance regressions promptly. It allows you to address these issues before they cause significant problems for your users. Maybe checkout the recent OpenTelemetry guidelines for Gen AI to observe how your system and LLM are performing live.

You can further improve your LLM-powered application by analyzing the user’s requests and responses generated by the LLM, especially those flagged by users as problematic or unexpected. These can be added to your collection of golden responses, constantly refining the process of evaluation. This helps your application evolve based on real-world interactions.

Addressing Data Privacy Concerns

Another important topic to keep in mind: data privacy. LLMs have access to a vast amount of text data, which makes them incredibly powerful. But this same power brings with it the responsibility of safeguarding sensitive information. If your application handles user data, you need to ensure that you’re handling it with utmost care, protecting it from unauthorized access and ensuring that you comply with relevant privacy regulations.

Think of data privacy as a trust contract. You, as the developer, are entrusted with safeguarding the sensitive information of your users. It’s your responsibility to implement measures that keep this data secure and prevent breaches or misuse.

Here are some key steps to address data privacy concerns in your LLM application:

  • Implement strong security measures: Use robust encryption methods to secure your application and data. Employ security best practices such as access controls, secure storage, and secure communication channels.
  • Stay aligned with data privacy regulations: Comply with relevant privacy regulations like GDPR, CCPA, and HIPAA. You might need to review your data handling policies and make necessary adjustments.
  • Ensure data anonymization: When working with sensitive data, always strive to anonymize or pseudonymize it to the fullest extent possible. You can utilize techniques like differential privacy, aggregation, or removing identifying details to protect user information (with Google Cloud Data Loss Prevention API for example).
  • Be transparent with users: Communicate clearly with your users about how you collect, use, and store their data. Offer users options to control their data, and provide mechanisms to update or delete their information if needed.

By prioritizing data privacy in your LLM application, you not only uphold ethical standards but also build trust with your users. Your users should be confident that their information is being handled with respect and care, encouraging long-term trust in your application.

Tailoring LLMs for Specific Business Goals

Remember that LLMs are tools, and the success of your LLM application ultimately hinges on aligning its capabilities with your unique goals and your target audience. So, how do you get the most out of an LLM in your business?

First, define your goals. What specific tasks can an LLM help you accomplish? What pain points are you trying to solve? Once you understand the big picture, you can break down those goals into actionable tasks that the LLM can potentially assist with.

Then, it’s time to find the right LLM for the job. Not all LLMs are created equal. Different models excel at specific tasks, have varying levels of language support, and even require different levels of computational resources. For example, if your business uses many different languages, you’ll want an LLM with multilingual support.

To select the best LLM for your needs, ask yourself:

  • What specific task does this LLM need to perform? Different LLMs excel at different tasks like text generation, summarization, or translation.
  • How does the LLM’s accuracy compare with the level of accuracy required for your application? The model needs to generate results with the appropriate level of precision for your goals.
  • How much computational power does it need to run this LLM? Consider your budget and available infrastructure when making this selection, when hosting the model on your own. A cloud hosted model might be better (and cheaper) depending on your usage patterns, and if you don’t want the hassle to handle your own infrastructure and GPUs.
  • What language capabilities does the LLM offer? Is the model good at the languages you need to use, or are there specific domains where the model is particularly strong? It’s not just about spoken languages, with code as well, some models maybe better dealing with a particular programming language than another one.

You can often find models with specialized skills. You may find, for example, a model trained on scientific papers if your work requires the processing of highly technical content, or a model trained on a particular field, such as text of laws, to be highly effective in that domain.

Once you’ve chosen your LLM, the next step could be fine-tuning, where you’d tailor the model to your specific needs. It’s like customizing a tool to do the exact job you need it to do. For example, imagine your application is helping people book vacations. You can train the model on a massive amount of vacation-related text data so it can accurately understand and respond to vacation-specific questions, making your application highly relevant for its intended purpose. But fine-tuning is not necessarily for the faint of heart, and can be complicated to do right.

While choosing and fine-tuning are critical steps, assessing potential risks is equally important. Think about potential unintended consequences. LLMs, for example, might not always be factual or accurate in their responses. You’ll need to find ways to manage those potential issues, often incorporating guardrails to mitigate potential harms or biases, or implementing techniques like Retrieval Augmented Generation to ground the model’s responses on your own data and documents.

Ultimately, you’ll want to make your application a tool that not only works reliably but also gives real value to your business. By understanding your business goals, choosing the right model, customizing it effectively, and understanding the potential risks, you’re on the right path to success!

Remember that this field is constantly changing! New capabilities are emerging, and existing models are getting even smarter and more efficient. This is an exciting time to be working with LLMs because the possibilities feel endless!

While it’s fantastic to get your application off the ground using the latest LLMs, it’s equally important to be open to continuous improvement. What’s great today may not be optimal in the future. The world of LLMs is one where ongoing development is key! Here are a few tips:

  • Embrace continuous learning. You should always be seeking out information about the newest developments in the field, how LLMs are being enhanced, and the impact those changes could have on your applications. Look out for improvements to existing models, new LLM models coming out, and fresh research.
  • Think ahead. What new features could you integrate in your application that take advantage of those advancements? Would your app benefit from a specific, task-oriented model that focuses on summarization, question answering, or code generation? Maybe there’s a model out there that will significantly boost performance and help you offer a smoother, more feature-rich experience for your users!
  • Prepare for evolution. Remember that LLMs aren’t static! Your app should be built with a framework for easy adaptation. Consider how you can adapt to model updates or new model releases in a structured way, perhaps by putting in place frameworks for incorporating new models seamlessly and managing prompt changes for various models, like LangChain4j (if you’re a Java developer) which offers higher-level abstractions and that allows you to switch models easily.

The landscape of LLMs is evolving rapidly. Stay up-to-date with the latest developments and ensure your applications can adapt, allowing you to unlock the full potential of LLMs for your business!