How I Built a Local, Multimodal Gemma 4 Visual Regression & Patch Agent: Closed-Loop Validation, Canvas Pixel Diffing, and Reproducible Benchmarks

Video Demo:

Visual display of the interactive Regression Loop application interface

Regression Loop for 'Split-slider', Side-by-side' and Pixel-diff-heatmap' visuals.

To complete the closed-loop developer experience, the frontend features a premium dashboard tab containing:

1. 
   Interactive Before/After Split Slider: Let developers scrub a visual slider side-by-side to compare the buggy UI with the expected fix state.


2. 
   Canvas-Computed Pixel Difference Heatmap: Leverages an HTML5 canvas to compare visual buffers in-browser. It maps changed pixels onto a semi-transparent red overlay and computes an alignment score:
   

const runPixelDiff = (imgA, imgB, canvas) => {

  const ctx = canvas.getContext('2d');

  const w = canvas.width, h = canvas.height;

  ctx.drawImage(imgA, 0, 0, w, h);

  const dataA = ctx.getImageData(0, 0, w, h);

  ctx.drawImage(imgB, 0, 0, w, h);

  const dataB = ctx.getImageData(0, 0, w, h);



  const diffImg = ctx.createImageData(w, h);

  let changedPixels = 0;

  for (let i = 0; i < dataA.data.length; i += 4) {

    const diffR = Math.abs(dataA.data[i] - dataB.data[i]);

    const diffG = Math.abs(dataA.data[i+1] - dataB.data[i+1]);

    const diffB = Math.abs(dataA.data[i+2] - dataB.data[i+2]);

    if (diffR > 45 || diffG > 45 || diffB > 45) {

      diffImg.data[i] = 255;   // Red highlight

      diffImg.data[i+1] = 0;

      diffImg.data[i+2] = 0;

      diffImg.data[i+3] = 160; // Transparency

      changedPixels++;

    }

  }

  ctx.putImageData(diffImg, 0, 0);

  const score = Math.max(0, 100 - (changedPixels / (w * h)) * 100);

  return score.toFixed(1);

};

📊 Evaluation & Empirical Benchmarks

To validate the agent's accuracy and reliability, we built an automated, reproducible benchmark framework (backend/benchmark.py). We evaluated the agent across 10 diverse test cases representing real-world frontend and backend bugs:

1. 
   CSS Overflow Bug: Container text overflowing without truncation controls.


2. 
   Z-Index Stacking Context: Modal overlay blocking standard content interactions.


3. 
   Flexbox Alignment Mismatch: Layout components failing to vertically align.


4. 
   Python AttributeError: Missing None checks on API response payloads.


5. 
   JS Event Handler Selectors: Target selectors mismatching DOM button bounds.


6. 
   CSS Contrast Violation: Low-contrast foreground and background colors.


7. 
   Sidebar Mobile Breakpoint: Layout breaks on smaller screen aspect ratios.


8. 
   Python Circular Dependency: Circular imports crash during service boot.


9. 
   SQL Injection Vulnerability: Missing parameter sanitization on user input queries.


10. 
    JS DOM Selector Mismatch: Target fields mismatching the email form input.

Benchmark Metrics Summary

• 
  Overall Agent Success Rate: 100.0% (10/10 cases resolved)


• 
  UI Bug Localization Accuracy: 100.0% (correct root cause selector tracing)


• 
  Git Apply Applicability Rate: 100.0% (clean, zero-hunk conflict applying)


• 
  AST / Syntax Validity Rate: 100.0% (zero syntax regression)


• 
  Average Analysis Latency: 0.90s
  


• 
  Average Patch Line Accuracy: 100.0% (identical alignment with human-engineered fixes)

🛠️ Reproducible Quick Start

You can run the entire agentic system and its benchmark suite locally in seconds using Mock Mode (no API keys required)!

1. Install Dependencies

# Clone the repository

git clone [email protected]:kanyingidickson-dev/Multimodal-Visual-Regression-Patch-Agent.git

cd Multimodal-Visual-Regression-Patch-Agent



# Set up virtual environment

python3 -m venv venv

source venv/bin/activate

pip install -r backend/requirements.txt

2. Compile Frontend Assets

cd frontend

npm install

npm run build

cd ..

3. Run Benchmark Suite

python3 backend/benchmark.py

This writes the test case directories, triggers the evaluation pipeline, and outputs a complete report inside examples/benchmark-cases/report.md.

4. Run FastAPI Server

python3 backend/app.py

Visit http://127.0.0.1:5000 to start visual regression testing interactively!

You can click 'Load Example' on Model settings for a quick demo launch and review.

🔮 The Road Ahead

This project shows what is possible when open multimodal models are coupled with deterministic validation sandboxes. By shifting the paradigm from "AI code review suggestions" to closed-loop visual agentic repair, we are paving the way for developers to resolve UI defects with full safety guarantees in seconds.

Built for the Gemma 4 Challenge:- demonstrating how open, multimodal models can empower developers with intelligent, visual-aware coding tools.

#ai #developertools #gemma4 #multimodal #agentic #patchvalidation #visualregression #opensource #devtools #coding #aiagents #gemma #gemma4challenge #hackathon #openai #google #developerexperience #visual-aware-coding #ai-agents #coding-assistant #visual-regression-patch-agent