Implementing Vision-Powered Chit-Chats with Robots: A GPT-4 Adventure 🤖👀

🖼️ GPT-4 and Images

To start, you'll need an of this project contains a Conversation class that does this (and other things too, more on this later).

📷 Taking pictures

In this example, during the conversation with the system, we're going to incorporate images in our prompt by taking a picture with the webcam at the beginning of the user's turn. In the you can find a version that continuously captures frames from a webcam, from a video or a Furhat robot.

✂️ Cut the prompt size

You'll quickly notice that your prompt will get too big, with slowed-down computation and increased prices. No good. To solve that, we thought of doing what's done with normal dialogue prompts: ask the LLM to summarise the first part of the conversation!

But we can't summarise images and dialogue together, a picture is worth a thousand words and our dialogue will virtually disappear in a sea of image descriptions. Remember when I told you that the Conversation class in the contains more details about it.

Here is the code that we used in the Conversation class.

def get_fr_summary(self) -> Tuple[List[Message], int]:

    """Summarise the frames and return the new messages and the number of frames removed."""

    # fr_buff_size is the max number of images (frames) in the prompt

    # fr_recap is the max number of frames to summarise

    # Assuming number of frames in prompt > fr_buff_size > fr_recap



    # Find the first frame and the last frame to summarise

    first_fr = None

    i = None

    for i, m in enumerate(self._messages):

        if m.is_frame():

            if first_fr is None:

                first_fr = i



        # Include at most fr_recap frames, and stop if we see a user message

        if first_fr is not None and (m.is_user() or i - first_fr >= self.fr_recap):

            break



    # Split the messages list

    before = self._messages[:first_fr]

    to_summarise = self._messages[first_fr:i]

    after = self._messages[i:]



    # Generate the summary

    prompt = [

        SystemMessage(

            "These are frames from a video. Summarise what's happening in the video in one sentence. "

            "The frames are preceded by a context to help you summarise the video. "

            "Summarise only the frames, not the context."

            "The images can be repeating, this is normal, do not point this out in the description."

            "Respond with only the summary in one sentence. This is very important. "

            "Do not include warnings or other messages."

        ).gpt_format(),

        *[b.gpt_format() for b in before],

        *[s.gpt_format() for s in to_summarise],

    ]

    summary = self.llm.query(prompt)



    # Generate the new message list with the summary

    messages = [

        *before,

        FSummaryMessage(summary),

        *after,

    ]



    return messages, i - first_fr

🔭 What now?

I hope this journey into combining GPT-4 with real-time visual input has sparked your curiosity! The possibilities are as vast as your imagination. Now armed with the knowledge to integrate large language models and live visual input, you can create a truly interactive and context-aware conversational experience. So, what are you waiting for? Dive into the code, explore the fascinating intersection of language and vision, and let your creativity run wild. The future of chatbots and social robots is not just text-based – it's a dynamic fusion of words and images, and you're at the forefront of it. We'll continue to work to improve this approach and explore new and exciting ways to make conversational agents better. Stay tuned!