Today I learned that I can show , toolkits are plugins that
"provide goose with tools (functions) it can call and optionally will load additional context into the system prompt (such as 'The Github CLI is called via gh and you should use it to run git commands'). Toolkits can do basically anything, from calling external APIs, to taking screenshots, to summarizing your current project."
How I Describe Toolkits 📱
Toolkits are like applications on your phone. At baseline, your phone can make calls, but you can download apps that extend its capabilities from playing games, to taking pictures, to listening to music. Similarly, with goose at baseline, you can manage plugins, start sessions, and manage versions. But you can add toolkits that enable you to:
- Take screenshots for debugging
- Interact with the GitHub CLI
- Manage Jira projects
- Summarize repositories
How to Use the Screen Toolkit 📸
Step 1: Install Goose ⚡
To install goose, run the following commands in your terminal:
brew install pipx
pipx ensurepath
pipx install goose-ai
Step 2: Start a Session 🚀
Start a goose session with the following command:
goose session start
Note: First, set an API key for your preferred LLM provider (like OpenAI, Anthropic, etc.). For example:
export OPENAI_API_KEY=your_api_key
# Or for other providers:
export ANTHROPIC_API_KEY=your_api_key
Step 3: Enable the Screen Toolkit ⚙️
You'll find a settings file at ~/.config/goose/profiles.yaml that configures how you want to use goose. In this case, you'll need to enable the screen toolkit. Here's an example of what it may look like:
default:
provider: openai
processor: gpt-4o
accelerator: gpt-4o-mini
moderator: truncate
toolkits:
- name: developer
requires: {}
- name: screen
requires: {}
Step 4: Prompt Goose to Take a Screenshot 📷
In natural language, you can prompt Goose to take a screenshot of your display with a line like:
"Take a screenshot of display (1)."
Goose will return the command it ran to capture the screenshot on your computer. It might look something like this:
screencapture -x -D 1 /tmp/goose_screenshot_a12e2775bdad4c55810e8f9812921731.jpg -f jpg
You can access the saved image by opening a new tab or terminal outside of a goose session and running the command:
open /tmp/goose_screenshot_a12e2775bdad4c55810e8f9812921731.jpg
Enable Goose to Update a UI via Screenshots 🎨
Now that we know how to capture screenshots with goose, let's use it to update a UI.
Step 1: Create a Simple UI 💻
We'll create an index.html file that renders three green boxes:
<!DOCTYPE html>
<html>
<head>
<style>
.container {
display: flex;
justify-content: center;
gap: 20px;
padding: 20px;
}
.box {
width: 100px;
height: 100px;
background-color: #4CAF50;
border-radius: 4px;
box-shadow: 0 2px 4px rgba(0,0,0,0.1);
}
</style>
</head>
<body>
<div class="container">
<div class="box"></div>
<div class="box"></div>
<div class="box"></div>
</div>
</body>
</html>
Save the file and open it in your browser. It should look like this:
Step 4: Verify the Result 🎉
Goose successfully updated the UI for me rendering three squares stacked vertically, with a pop of pink in the middle.
to discover more possibilities.
As I was writing this blog post, I realized there are opportunities for us to add new capabilities like screen recording to enable goose to handle keyframe animations and transformations.
Join Our Community! 🤝
The Goose Open Source Community is growing quickly, and I invite you join the fun by:
- Joining our
Can't wait to see you there!
SOCIAL SHARE CARD GENERATOR