The Problem Nobody Talks About
Every election cycle in India, candidates are not easily accessible to many so they don't the many information about candidate's they are voting for.
What I Built
Candidate Compliance Agent is an AI agent that lets anyone query Tamil Nadu election candidate affidavits in plain English or Tamil. Ask it about assets, criminal records, income declarations, education — it answers from the actual documents.
Live demo: https://affidavit-rag-frontend.vercel.app
The Technical Challenge
This wasn't a clean dataset project. The raw materials were:
Scanned PDFs (~6MB each)
Mixed Tamil and English text
Stamp paper backgrounds with notary seals overlaid on text
Inconsistent formatting across candidates
Architecture Overview
React Frontend (Vercel)
↓
Flask Backend (Railway)
↓
Hindsight Memory (recall past context)
↓
Intent Router
↓
Structured Query OR Semantic RAG
↓
Groq LLM (Llama 3.3 70B)
↓
Hindsight Memory (retain new context)
↓
Response + Follow-up Questions
Technical Feature Implemnentaion Walkthrough
OCR Pipeline
The first challenge was extracting text from scanned PDFs. I used Tesseract with Tamil + English language models:
python
text = pytesseract.image_to_string(
image,
lang="eng+tam",
config="--psm 6"
)
--psm 6 treats each page as a uniform text block — critical for structured affidavit forms.
Each PDF was converted to images at 300 DPI using pdf2image, then OCR'd page by page. Output was stored as JSON with metadata:
{
"text": "...",
"metadata": {
"candidate": "MKStalin",
"party": "DMK",
"constituency": "KOLATHUR",
"page": 3
}
}
Semantic Document Building
The biggest architectural decision was how to chunk the data.
Naive approach: Split each page into 500-token chunks with overlap. This gives you 600+ noisy chunks where assets, criminal records, and education are all mixed together.
Better approach: Build one semantic document per topic per candidate.
Each candidate gets 6 documents:
Identity and contact
Education
Criminal cases (structured)
Criminal detail (raw OCR text)
Income tax
Assets and liabilities
def build_docs(candidate, raw_pages):
docs = []
docs.append({"section": "criminal_cases", "text": f"""
Candidate: {name}
Party: {party}
Section: Criminal Cases
Pending Cases: {pending}
Convicted Cases: {convicted}
Summary: {"No criminal record" if pending == 0 else f"{pending} pending"}
"""})
# ... repeat for each section
return docs
35 candidates × 6 sections = 210 clean semantic documents instead of 600+ noisy chunks.
Result: When someone asks "does certain candidate have criminal cases?" the retrieval hits the criminal_cases document directly instead of a random page slice.
Embeddings and Vector Store
I used paraphrase-multilingual-MiniLM-L12-v2 for embeddings — a sentence transformer model that handles Tamil and English in the same vector space.
model = SentenceTransformer("paraphrase-multilingual-MiniLM-L12-v2")
embeddings = model.encode(texts, batch_size=32).tolist()
collection.add(
documents=texts,
embeddings=embeddings,
metadatas=[{
"candidate": d["candidate"],
"party": d["party"],
"section": d["section"],
} for d in all_docs],
ids=[f"{d['candidate']}_{d['section']}" for d in all_docs]
)
Intent Router
there are 5 intent category based on that the query is passed through
def detect_intent(query):
q = query.lower()
for party in PARTIES:
if party in q:
return "party"
for pattern in EDUCATION_PATTERNS:
if re.search(pattern, q):
return "education"
for word in SEMANTIC_KEYWORDS:
if word in q:
return "semantic"
return "unknown"
Hindsight Memory Integration
Hindsight is a persistent memory service that stores and recalls context across sessions. When a user comes back the next day and asks a follow-up question, the agent remembers what they asked before.
from hindsight_client import Hindsight
client = Hindsight(
base_url="https://api.hindsight.vectorize.io",
api_key=os.getenv("HINDSIGHT_API_KEY")
)
def recall_memory(session_id, query):
result = client.recall(
bank_id="candidate-rag",
query=f"Session: {session_id}\nQuestion: {query}"
)
return "\n".join([m.text for m in result.results])
def retain_memory(session_id, memory):
client.retain(
bank_id="candidate-rag",
content=memory,
metadata={"session_id": session_id}
)
USING THE MEMORY STORED IN HINDSIGHT
1. Recall past context
past_memory = recall_memory(session_id, query)
2. Inject into RAG prompt
prompt = f"""
Previous context: {past_memory}
Context from affidavits: {retrieved_context}
Question: {query}
"""
3. Get answer
answer = ask_groq(prompt)
4. Store this exchange
retain_memory(session_id, f"User asked: {query}. Answer: {answer[:200]}")
Why this matters: A voter researching a candidate can ask multiple questions across multiple sessions and get progressively better, context-aware answers.
Contextual Follow-up Questions
After every answer, the LLM generates 4 follow-up questions based on the query, retrieved context, and memory:
ANSWER:
Poornima has no pending criminal cases.
FOLLOW_UP_QUESTIONS:
- What are Poornima's total declared assets?
- What is Poornima's declared annual income?
- What is Poornima's educational qualification?
These appear as clickable buttons — clicking fills the input box directly.
Deployment
Backend: Flask + gunicorn on Railway
Frontend: React on Vercel
Vector DB: ChromaDB (persistent, committed to GitHub)
LLM: Groq API (Llama 3.3 70B Versatile)
Memory: Hindsight
What I Learned
OCR quality determines everything downstream. Bad OCR produces bad embeddings. The --psm 6 config change improved Tamil extraction significantly.
Semantic chunking beats token chunking. 210 topic-focused documents retrieve better than 600+ random page slices.
Persistent memory changes the interaction model. Without Hindsight, every conversation starts cold. With it, the agent builds understanding over time.
Real-world data is always messier than tutorials suggest. Stamp paper backgrounds, notary seal overlays, mixed scripts — none of this appears in RAG tutorials.
What's Next
Scale to all 234 Tamil Nadu constituencies (~4500 PDFs)
Add proactive alerts when new affidavits are filed
Support voice queries in Tamil
SOCIAL SHARE CARD GENERATOR