🎯
Architecture Essentials
Probabilistic Automation: Moving beyond rigid rules
3-Tier Integration: Utility -> Self-Healing -> Agentic
Safety First: Run LLMs locally (Ollama) for privacy
Python: The glue code for the AI era
As automation architects, we are used to rigid structures. We build frameworks based on predictability: If this element exists, click it. If this assertion fails, stop.
But the introduction of Large Language Models (LLMs) changes the fundamental way of our work. We are moving from deterministic automation (rules-based) to probabilistic automation (inference-based).
Mindset Shift
Mindset Shift
This isn't about asking ChatGPT to write regex. It's about fundamentally restructuring your framework to be "intelligent".
🧠
For an automation architect, this isn't just about asking ChatGPT to write a regex for us. It is about fundamentally re-structuring our framework to be "intelligent"—capable of understanding intent, healing itself, and analyzing complex failures.
Here is what an LLM actually is in our context, and how we should architect it into our Python ecosystem.
What is an LLM (In Our World)?
Let's keep aside the "wikipedia" definitions for a moment. In the context of test automation, an LLM is a semantic engine. (atleast thinking in this term helps to restructure our frameworks)
Traditional automation tools (Selenium, Playwright) interact with the syntax of an application (DOM, ID, XPath). They don't understand what a "Login Button" is; they only know it as #btn-login.
An LLM acts as a translation layer that understands the semantics (meaning) of the application. It can look at a raw HTML dump and understand, "Ah, this is a credit card form, and that obscure div is likely the submit button."
For an architect, an LLM is a new component in our system design—like a database or a message queue—that processes unstructured data (logs, DOM, user stories) and returns structured actions.
Architectural Strategy: How to Integrate LLMs
Do not just "sprinkle AI" everywhere. As an architect, we need a strategy. I recommend a three-tiered integration approach for Python frameworks.
Tier 1: The "Smart" Utility Layer (Low Risk)
Start by adding an LLM service class to our utils folder. This layer does not execute tests but supports them.
Test Data Generation: Instead of using static JSON files or basic Faker libraries, use an LLM to generate context-aware edge cases (e.g., "Generate 5 valid addresses in Germany that would fail a regex check due to special characters").
Log Analysis: When a test fails, pass the traceback and the last 10 lines of the system log to a local model (like Ollama/Llama 3). Have it append a "Root Cause Hypothesis" to our HTML report.
Tier 2: The Self-Healing Driver (Medium Risk)
This is where we wrap our core driver (Selenium/Playwright) with intelligence.
The Problem: The UI changes. The ID #submit becomes #submit-v2. our script fails. The LLM Solution:
- Catch the NoSuchElementException.
- Capture the current DOM state (truncated to fit context window).
- Send the DOM + the original locator to the LLM.
- Prompt: "The element #submit is missing. Based on the current HTML, what is the new most likely selector for the 'Submit' button? Return only the selector."
- Retry the action with the new selector.
Tier 3: The Agentic Framework (High Ambition)
This uses libraries like LangChain or AutoGen. Instead of writing linear test scripts, we write "Goals."
Goal: "Verify the checkout flow for a guest user."
Agent: The agent spawns a browser, observes the screen, decides which Python function to call (click_element, enter_text), and loops until the goal is met or it gets stuck.
Python Implementation: A "Self-Healing" Example
Let's look at a concrete implementation for Tier 2 using Python. We will create a decorator that we can wrap around our page object methods.
Prerequisites: Python 3.10+, openai library (or requests if using a local Ollama server)
import functools
from openai import OpenAI
from selenium.common.exceptions import NoSuchElementException
# Initialize client (Point this to localhost:11434 for Ollama if we want offline privacy)
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
def self_healing(func):
"""
A decorator that attempts to heal a failed element interaction.
"""
@functools.wraps(func)
def wrapper(*args, **kwargs):
try:
return func(*args, **kwargs)
except NoSuchElementException as e:
print(f"Element missing in {func.__name__}. Attempting to heal...")
# Assuming 'self' is the first arg and has a 'driver' property
driver = args[0].driver
page_source = driver.page_source[:2000] # Truncate for token limits
# Ask the LLM for help
prompt = f"""
I tried to find an element but failed.
The intended action was inside function: '{func.__name__}'.
Here is a snippet of the page HTML:
{page_source}
Identify the CSS selector that most likely represents the element intended by '{func.__name__}'.
Return ONLY the CSS selector string.
"""
response = client.chat.completions.create(
model="llama3", # Using a local model
messages=[{"role": "user", "content": prompt}]
)
new_selector = response.choices[0].message.content.strip()
print(f"LLM suggested new selector: {new_selector}")
# Retry finding the element with the new selector
# Note: we would need to adapt our function to accept a dynamic locator override
return driver.find_element("css selector", new_selector)
return wrapper
# Usage in our Page Object
class LoginPage:
def __init__(self, driver):
self.driver = driver
@self_healing
def click_login(self):
# Even if this ID changes, the LLM might find the new button based on class or text
return self.driver.find_element("id", "old-login-id").click()
Best Practices for the Architect
Local First: For sensitive corporate environments, do not send DOM data to public APIs (OpenAI/Claude). Use Ollama or LM Studio to host models like Llama 3 or Mistral locally on our machine or an internal server. This solves the data privacy hurdle immediately.
Context is King: An LLM is only as good as the data we feed it. Don't just send the error message; send the DOM snippet, the test intent, and the previous successful run logs if possible.
Human in the Loop: Never let an LLM "auto-commit" fixes to our code repository. Use the LLM to generate a "Patch Suggestion" file that a human engineer must review and approve.
🛠️
Conclusion
The role of the Automation Architect is shifting from "maintaining the framework" to "training the assistant." By integrating LLMs via Python, we aren't just making tests less flaky; we are building a system that understands our application almost as well as we do.
Start small. Implement the "Log Analyzer" today, and work your way up to the self-healing driver. And as always, Get In Touch! if any challenges faced
SOCIAL SHARE CARD GENERATOR