Counting the number of Tokens sent to a LLM in Go (part 2)

. The code seems to be very new but it’s “kind-of” working for me.

Implementation

First of, the code accesses Hugginface to get the list of all the “tokenizers” which go with a LLM, so the user should have a HF token. So I put my token in an .env file as shown.

HF_TOKEN="your-huggingface-token"

Then using the example provided in the following page (

If everything goes fine, the next step is to download the “tokenizer” file locally (refer to the Github repo’s explanations) and then a dialog box is shown to chose the text file with the content which is to be evaluated in terms of the number of tokens.

So far, I have asked access to Meta LLama and Google “google/gemma-2–2b-it” models and am awaiting access to be granted.

google/gemma-2-2b-it:

        repo=google/gemma-2-2b-it

panic: request for metadata from "https://huggingface.co/google/gemma-2-2b-it/resolve/299a8560bedf22ed1c72a8a11e7dce4a7f9f51f8/tokenizer_config.json" failed with the following message: "403 Forbidden"

Conclusion

I think being on the right path to achieve what I intended to get, a Golang programme which is able to determine the number of tokens is a user’s query sent to a LLM.

The only aim of this project is to learn the internal system behind determination of the number of tokens in queries against a variety of LLMs and to discover how they are calculated.

Thanks for reading and open to comments.

And till the final conclusion, stay tuned… 🧪