In this article, we’ll explore another great task where Large Language Models shine: entity and data extraction. LLMs are really useful beyond just mere chatbots (even smart ones using Retrieval Augmented Generation).

Let me tell you a little story of a handy application we could build, for wannabe Instagram influencers!

Great Instagram hashtags, thanks to LLMs

When posting Instagram pictures, I often struggle with finding the right hashtags to engage with the community. Large Language Models are pretty creative, and they’ve certainly seen a bunch of Instagram pictures with their descriptions.

Generative AI use cases are usually about running large language models somewhere in the cloud. However, with the advent of smaller models and open models, you can run them locally on your machine, with projects like llama.cpp or Ollama.

And what about in the browser? With MediaPipe and TensorFlow.js, you can train and run small neural networks for tons of fun and useful tasks (like recognising hand movements through the webcam of your computer), and it’s also possible to run Gemma 2B and even 7B models.

In a rencent article, we talked about text classification using Gemini and LangChain4j.

A typical example of text classification is the case of sentiment analysis.

In my LangChain4j-powered Gemini workshop, I used this use case to illustrate the classification problem:

ChatLanguageModel model = VertexAiGeminiChatModel.builder()
    .project(System.getenv("PROJECT_ID"))
    .location(System.getenv("LOCATION"))
    .modelName("gemini-1.5-flash-001")
    .maxOutputTokens(10)
    .maxRetries(3)
    .build();

PromptTemplate promptTemplate = PromptTemplate.from("""
    Analyze the sentiment of the text below.
    Respond only with one word to describe the sentiment.

    INPUT: This is fantastic news!
    OUTPUT: POSITIVE

    INPUT: Pi is roughly equal to 3.14
    OUTPUT: NEUTRAL

    INPUT: I really disliked the pizza. Who would use pineapples as a pizza topping?
    OUTPUT: NEGATIVE

    INPUT: {{text}}
    OUTPUT:
    """);

Prompt prompt = promptTemplate.apply(
    Map.of("text", "I love strawberries!"));

Response<AiMessage> response = model.generate(prompt.toUserMessage());

System.out.println(response.content().text());

I used a PromptTemplate to craft the prompt, with a {{text}} placeholder value to analyze the sentiment of that particular text.

Certain models like Gemini are multimodal. This means that they accept more than just text as input. Some models support text and images, but Gemini goes further and also supports audio, video, and PDF files. So you can mix and match text prompts and different multimedia files or PDF documents.

Until LangChain4j 0.32, the models could only support text and images, but since my PR got merged into the newly released 0.33 version, you can use all those files with the LangChain4j Gemini module!

Generative AI has potential applications far beyond chatbots and Retrieval Augmented Generation. For example, a nice use case is: text classification.

I had the chance of meeting some customers and prospects who had the need for triaging incoming requests, or for labeling existing data. In the first case, a government entity was tasked with routing citizen requests to access undisclosed information to the right governmental service that could grant or reject that access. In the second case, a company needed to sort out tons of existing internal documents that were not properly organized, and they wanted to quickly start better structuring this trove of information, by labelling each of these docs into different categories.

LangChain4j 0.32.0 was released yesterday, including my pull request with the support for lots of new Gemini features:

• JSON output mode, to force Gemini to reply using JSON, without any markup,
• JSON schema, to control and constrain the JSON output to comply with a schema,
• Response grounding with Google Search web results and with private data in Vertex AI datastores,
• Easier debugging, thanks to new builder methods to log requests and responses,
• Function calling mode (none, automatic, or a subset of functions),
• Safety settings to catch harmful prompts and responses.

Let’s explore those new features together, thanks to some code examples! And at the end of the article, if you make it through, you’ll also discover 2 extra bonus points.

Prelude

As I’m focusing a lot on Generative AI, I’m curious about how things work under the hood, to better understand what I’m using in my gen-ai powered projects. A topic I’d like to focus on more is: vector embeddings, to explain more clearly what they are, how they are calculated, and what you can do with them.

A colleague of mine, André, was showing me a cool experiment he’s been working on, to help people prepare an interview, with the help of an AI, to shape the structure of the resulting final article to write.

Read more...

A few months ago, I shared an article about what I called Java functional builders, inspired by an equivalent pattern found in Go. The main idea was to have builders that looked like this example:

LanguageModel languageModel = new LanguageModel(
    name("cool-model"),
    project("my-project"),
    temperature(0.5),
    description("This is a generative model")
);

Compared to the more tranditional builder approach:

• You’re using the new keyword again to construct instances.
• There’s no more build() method, which felt a bit verbose.

Compared to using constructors with tons of parameters:

The happy users of Gemini Advanced, the powerful AI web assistant powered by the Gemini model, can execute some Python code, thanks to a built-in Python interpreter. So, for math, logic, calculation questions, the assistant can let Gemini invent a Python script, and execute it, to let users get a more accurate answer to their queries.

But wearing my Apache Groovy hat on, I wondered if I could get Gemini to invoke some Groovy scripts as well, for advanced math questions!

The latest release of LangChain4j (version 0.31) added the capability of grounding large language models with results from web searches. There’s an integration with Google Custom Search Engine, and also Tavily.

The fact of grounding an LLM’s response with the results from a search engine allows the LLM to find relevant information about the query from web searches, which will likely include up-to-date information that the model won’t have seen during its training, past its cut-off date when the training ended.