How to deploy InternVL2-2B in the Cloud?

Performance

Image Benchmarks

Grounding Benchmarks

Step 2: Create a GPU Node (Virtual Machine)

GPU Nodes are NodeShift's GPU Virtual Machines, on-demand resources equipped with diverse GPUs ranging from H100s to A100s. These GPU-powered VMs provide enhanced environmental control, allowing configuration adjustments for GPUs, CPUs, RAM, and Storage based on specific requirements.

We will use 1x H100 SXM GPU for this tutorial to achieve the fastest performance. However, you can choose a more affordable GPU with less VRAM if that better suits your requirements.

Step 4: Select Authentication Method

There are two authentication methods available: Password and SSH Key. SSH keys are a more secure option. To create them, please refer to our official documentation.

After choosing the image, click the 'Create' button, and your Virtual Machine will be deployed.

Step 7: Connect to Jupyter Notebook

Once your GPU Virtual Machine deployment is successfully created and has reached the 'RUNNING' status, you can navigate to the page of your GPU Deployment Instance. Then, click the 'Connect' Button in the top right corner.

Now open Python 3(pykernel) Notebook.

Next, If you want to check the GPU details, run the command in the Jupyter Notebook cell:

!nvidia-smi



Step 10: Install the Required Packages

Run the following command in the Jupyter Notebook cell to install the required packages:

!pip install transformers safetensors decord



!pip install timm

Step 12: Load the Model

Run the following model code in the Jupyter Notebook to load the model:

import torch

from transformers import AutoTokenizer, AutoModel



# Specify the model path

model_path = "OpenGVLab/InternVL2-2B"



# Load tokenizer and model with GPU acceleration

tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)

model = AutoModel.from_pretrained(

    model_path,

    torch_dtype=torch.bfloat16,  # For memory-efficient usage

    low_cpu_mem_usage=True,

    use_flash_attn=True,

    trust_remote_code=True

).eval().cuda()  # Load the model onto GPU

Step 14: Handling Images in InternVL2-2B

Run the following model Handling Images code in the Jupyter Notebook to load the model:

from PIL import Image

import torchvision.transforms as T

import torch



def preprocess_image(image_path):

    transform = T.Compose([

        T.Resize((448, 448)),

        T.ToTensor(),

        T.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225))

    ])

    image = Image.open(image_path).convert("RGB")



    # Apply transformations and move to GPU with BFloat16 precision

    return transform(image).unsqueeze(0).to(torch.bfloat16).cuda()



# Load and preprocess the image

image = preprocess_image("./NodeShift.png")



# Ask the model to describe the image

question = "<image>\nPlease describe the image shortly."

response = model.chat(tokenizer, image, question, generation_config)



print(f"User: {question}\nAssistant: {response}")

Upload the image in Jupyter Notebook from the upload button.

Upload the image in Jupyter Notebook from the upload button.

Step 15: Check Output

Below is Image 1, which we are uploading into the Jupyter Notebook. Refer to Image 2 below for the output.

Image 2

Step 16: Batch Inference and Video Support

For batch processing of multiple images or video frames, load the following model code in Jupyter Notebook:

import torch

from PIL import Image

import torchvision.transforms as T

from decord import VideoReader, cpu



# Define the image preprocessing function for video frames

def preprocess_frame(frame):

    transform = T.Compose([

        T.Resize((448, 448)),

        T.ToTensor(),

        T.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225))

    ])

    # Apply the transformation and move the tensor to GPU

    return transform(frame).unsqueeze(0).to(torch.bfloat16).cuda()



# Load video frames and preprocess them

def load_video_frames(video_path, num_segments=8):

    vr = VideoReader(video_path, ctx=cpu(0))

    indices = [int(i * len(vr) / num_segments) for i in range(num_segments)]

    frames = [Image.fromarray(vr[idx].asnumpy()).convert("RGB") for idx in indices]



    # Preprocess each frame and stack them into a single tensor

    preprocessed_frames = [preprocess_frame(frame) for frame in frames]

    return torch.cat(preprocessed_frames, dim=0)  # Combine frames along batch dimension



# Load and preprocess the video frames

video_frames = load_video_frames("./Horse.mp4")



# Ask the model to describe the video

question = "Describe the actions in the video."

response = model.chat(tokenizer, video_frames, question, generation_config)



print(f"User: {question}\nAssistant: {response}")

Upload the video in Jupyter Notebook from the upload button.

Step 17: Check Output

Example 1

Video Link:

Example 2

Video Link:

Conclusion

InternVL2-2B is a groundbreaking open-source model from OpenGVLab

that brings state-of-the-art AI capabilities to developers and researchers. Following this step-by-step guide, you can quickly deploy InternVL2-2B on a GPU-powered Virtual Machine with NodeShift, harnessing its full potential. NodeShift provides an accessible, secure, affordable platform to run your AI models efficiently. It is an excellent choice for those experimenting with InternVL2-2B and other cutting-edge AI models.

For more information about NodeShift: