Getting started with Google's Multi-modal "Gemini Pro Vision" LLM with Javascript for Beginners

// This imports the new Gemini LLM
import { GoogleGenerativeAI } from "@google/generative-ai";

// This imports fs from node.js, which we 
// will use to read the image file
import * as fs from "fs";

// This helps connect to our .env file
import * as dotenv from "dotenv";
dotenv.config();

const template = `Task: Analyze images of nutritional facts from various products, which feature diverse designs and include detailed nutritional information typically found on food packaging.

Step 1: Image Recognition and Verification

Confirm if each image is a nutritional facts label.
If the image is not a nutritional facts label, return: "Image not recognized as a nutritional facts label."
Step 2: Data Extraction and Nutritional Analysis

Extract and list the key nutritional contents from the label, such as calories, fats, carbohydrates, proteins, vitamins, minerals and other nutritional details.
Step 3: Reporting in a Strict Two-Sentence Format

First Sentence (Nutritional Content Listing): Clearly list the main nutritional contents. Example: "This product contains 200 calories, 10g of fat, 5g of sugar, and 10g of protein per serving."
Second Sentence (Diet-Related Insight): Provide a brief, diet-related insight based on the nutritional content. Example: "Its high protein and low sugar content make it a suitable option for muscle-building diets and those managing blood sugar levels."
Note: The response must strictly adhere to the two-sentence format, focusing first on listing the nutritional contents, followed by a concise dietary insight.
`;

// Above we created a template variable that 
// contains our detailed instructions for the LLM

// Below we create a new instance of the GoogleGenerativeAI
// class and pass in our API key as an argument
const genAI = new GoogleGenerativeAI(process.env.GOOGLE_API_KEY);

// This function takes a path to a file and a mime type
// and returns a generative part (specific object structure) that 
// can be passed to the generative model
function fileToGenerativePart(path, mimeType) {
  return {
    inlineData: {
      data: Buffer.from(fs.readFileSync(path)).toString("base64"),
      mimeType,
    },
  };
}

// We create the model variable by calling the getGenerativeModel method
// on our genAI instance and passing in an object with the model name
// as an argument. Since we want to interact with images, we use the
// gemini-pro-vision model.
const model = genAI.getGenerativeModel({ model: "gemini-pro-vision" });

// We create our image variable by calling the fileToGenerativePart function
// and passing in the path to our image file and the mime type of the image.
const image = [fileToGenerativePart("./pepsi.jpeg", "image/jpeg")];

// We call the generateContent method on our model and pass in our template
// and image variables. This returns a promise, which we await and assign
// to the result variable. We then await the response property of the result
// variable and assign it to the response variable. Finally, we await the text
// method of the response variable and assign it to the text variable.
const result = await model.generateContent([template, ...image]);
const response = await result.response;
const text = response.text();

// Log the text variable to the console.
console.log(text);