Visualize PaLM-based LLM tokens
As I was working on tweaking the Vertex AI text embedding model in LangChain4j, I wanted to better understand how the textembedding-gecko model tokenizes the text, in particular when we implement the Retrieval Augmented Generation approach.
The various PaLM-based models offer a computeTokens endpoint, which returns a list of tokens (encoded in Base 64) and their respective IDs.
Note: At the time of this writing, there’s no equivalent endpoint for Gemini models.
Read more...