NitroGen — Vision-to-Action Game AI

NitroGen is an open research project from the MineDojo ecosystem exploring a simple but powerful idea:

An AI can learn to play games by looking at the screen and imitating human actions — without access to the game engine, APIs, or internal state.

This repository provides a reference implementation of a vision-to-action game-playing agent trained via imitation learning.

Key Idea

Instead of reinforcement learning and reward engineering, NitroGen uses behavior cloning from real human gameplay videos.

The model learns a direct mapping:

screen pixels → neural network → controller actions

This allows the same agent architecture to work across many different games.

What Problems Does NitroGen Solve?

Traditional game AI often requires:

• game engine access


• internal state or memory reading


• custom APIs or SDKs


• handcrafted reward functions

NitroGen avoids all of the above.

Advantages

• Engine-agnostic


• Game-agnostic


• No reward functions


• Faster experimentation than RL


• Human-like behavior

How NitroGen Works (High Level)

1. Collect gameplay videos from real human players
   


2. Extract player actions from controller overlays
   


3. Train a vision-based neural network via imitation learning
   


4. Predict the next action given the current frame
   

The result is a general game-playing agent that operates purely from visual input.

Use Cases

Legitimate use cases include:

• Game AI research


• Imitation learning experiments


• Multi-game agents


• Automated game testing (QA)


• Accessibility tools


• Embodied AI research


• Education and ML courses

⚠️ This project is not intended for cheating, farming, or online exploitation.

Repository Structure

nitrogen/

├── models/        # Vision-to-action models

├── datasets/      # Gameplay datasets

├── envs/          # Game wrappers / environments

├── scripts/       # Training and evaluation scripts

├── configs/       # Experiment configurations

└── checkpoints/   # Pretrained weights (if available)

Installation

Requirements

• Python 3.9+


• Linux or macOS recommended


• GPU strongly recommended

Setup

Clone the repository: https://github.com/MineDojo/NitroGen

git clone https://github.com/minedojo/nitrogen.git

cd nitrogen

Create a virtual environment:

python -m venv venv

source venv/bin/activate

Install dependencies:

pip install -r requirements.txt

Running a Pretrained Model

If pretrained checkpoints are available:

python scripts/run_agent.py \

  --config configs/eval.yaml \

  --checkpoint checkpoints/nitrogen.pt

The agent will:

• receive game frames


• predict controller actions


• interact with the environment in real time

Training

Minimal training example:

python scripts/train.py \

  --config configs/train.yaml

Key parameters to tune:

• frame resolution


• action space


• sequence length


• dataset quality

Training is significantly more stable than reinforcement learning approaches.

Limitations

• Performance depends heavily on data quality


• No long-term planning by default


• Limited temporal reasoning


• Not optimized for competitive or online play

NitroGen is a foundation, not a finished product.

Why This Project Matters

NitroGen provides:

• A clean reference implementation of vision-to-action agents


• A scalable alternative to reinforcement learning


• A practical starting point for general game AI research

If you want to experiment with game AI without touching the game engine, this project is a strong base.

Get Involved

If this project is useful to you:

• ⭐ Star the repository


• 🧪 Run experiments


• 🐛 Open issues


• 🔧 Build on top of it