Algolia MCP Server Challenge Submission

Whispers - A Contextual Voice Memory System

What I Built

Whispers is a voice-first journaling application that transforms spoken thoughts into searchable, contextual memories. Users speak naturally into their microphone, and the system captures, processes, and indexes their reflections with semantic understanding. The core innovation is using Algolia MCP Server to power intelligent search that goes beyond keyword matching—it understands context, emotional states, and temporal patterns in your personal narrative.

This isn't just a search engine for text. It's a second brain that remembers not just what you said, but when you said it, how you felt, and what patterns emerge across your thoughts over time.

Demo

🎥 Video Demo:

https://drive.google.com/file/d/1RHyqpW434EeTGdP6xMRYbZCfifNatZd7/view?usp=sharing

GitHub Repository

The complete source code is available at: (https://github.com/VaishakhVipin/whispers-final)

Key files demonstrating Algolia MCP integration:

• 
  backend/services/gemini.py - MCP search orchestration and query decomposition


• 
  backend/routes/stream.py - Algolia indexing and filtered search endpoints


• 
  frontend/src/components/SearchInterface.tsx - Natural language search interface


• 
  backend/services/algolia.py - Algolia MCP client implementation

How I Utilized the Algolia MCP Server

The Algolia MCP Server is the backbone of Whispers' contextual memory system. Here's how it transforms natural language queries into intelligent, filtered search results:

1. Structured Data Indexing with Rich Metadata

Each journal entry is indexed with comprehensive metadata that enables sophisticated filtering:

entry = {

    "user_id": user.id,           # User isolation

    "session_id": session_id,     # Session grouping

    "date": date,                 # Temporal filtering

    "timestamp": timestamp,       # Precise timing

    "title": title,               # Semantic search

    "summary": summary,           # Contextual understanding

    "tags": tags,                 # Emotional/topic classification

    "text": text,                 # Full content search

    "is_from_prompt": is_from_prompt  # Prompt-driven vs free-form

}

2. Gemini-Powered Query Decomposition

When users ask questions like "When was I stuck?" or "What were my creative ideas last month?", Gemini breaks these into searchable components:

def mcp_search(query, user_id=None):

    # Step 1: Extract search terms and determine intent

    extraction_prompt = (

        "Extract the most relevant search terms and provide a helpful response. "

        "Return a JSON object with: "

        "1. 'is_search': 'yes' if this is a search query, 'no' otherwise "

        "2. 'search_terms': array of specific search terms to use "

        "3. 'gemini_response': a brief, helpful response about what you're looking for "

        f"User query: {query}"

    )



    # Step 2: Query Algolia with user-specific filters

    for term in search_terms:

        request_body = {

            "indexName": ALGOLIA_INDEX_NAME,

            "query": term,

            "hitsPerPage": 10,

            "filters": f"user_id:{user_id}"  # Critical: user data isolation

        }

3. Contextual Relevance Scoring

Results are ranked by semantic relevance, not just keyword frequency:

def calculate_relevance(hit):

    relevance_score = 0

    for term in search_terms:

        term_lower = term.lower()

        if term_lower in hit.get("title", "").lower():

            relevance_score += 3  # Title matches are most important

        if term_lower in hit.get("summary", "").lower():

            relevance_score += 2  # Summary matches are important

        if any(term_lower in tag.lower() for tag in hit.get("tags", [])):

            relevance_score += 1  # Tag matches are good

    return relevance_score

4. Real-World Search Examples

Query: "When did I feel burnt out?"

• 
  Gemini Decomposition: ["burnt", "out", "burnout", "exhausted"]
  


• 
  Algolia Filter: user_id:123 AND (burnt OR out OR burnout OR exhausted)
  


• 
  Result: Entries tagged with "burnout", "stress", or containing emotional context

Query: "What were my app ideas last month?"

• 
  Gemini Decomposition: ["app", "ideas", "startup", "project"]
  


• 
  Algolia Filter: user_id:123 AND date:2024-06* AND (app OR ideas OR startup OR project)
  


• 
  Result: Creative entries from June with relevant tags

Key Technical Achievements

Contextual Memory Recall

• 
  Semantic Understanding: Queries like "when I was struggling" find entries with emotional context, not just the word "struggling"


• 
  Temporal Intelligence: "Last week" automatically filters to recent entries


• 
  Pattern Recognition: Identifies recurring themes across multiple entries

Privacy-First Architecture

• 
  User Isolation: Every search is filtered by user_id ensuring complete data separation


• 
  Secure Indexing: No cross-user data leakage in the Algolia index


• 
  Audit Trail: All search queries are logged for transparency

Performance Optimization

• 
  Sub-200ms Search: Algolia's distributed search infrastructure delivers instant results


• 
  Smart Caching: Frequently accessed patterns are cached for faster retrieval


• 
  Efficient Filtering: User-specific filters reduce search space and improve performance

Key Takeaways

1. MCP Enables Contextual Search: Traditional search engines match keywords. MCP with Gemini enables understanding of intent, emotion, and temporal context.




2. Structured Data Powers Intelligence: Rich metadata (tags, dates, user context) transforms simple text search into intelligent memory recall.




3. User Isolation is Critical: Multi-tenant applications require careful filter design to prevent data leakage while maintaining search performance.




4. Natural Language Queries Need Decomposition: Complex questions require breaking down into searchable components while preserving semantic meaning.




5. Relevance Scoring Matters: Beyond simple keyword matching, contextual relevance scoring ensures users find the most meaningful memories.

Technical Stack

Voice Processing:

• AssemblyAI Universal Streaming for real-time transcription


• WebSocket for low-latency audio streaming

AI & Search:

• Google Gemini for query decomposition and content analysis


• Algolia MCP Server for contextual search and filtering


• FastAPI for backend orchestration

Data Architecture:

• Supabase for user authentication and session management


• Algolia for search indexing with rich metadata


• React + TypeScript for responsive frontend

Deployment:

• Vercel for frontend hosting


• Vercel Functions for serverless backend


• Environment-based security configuration

What's Next

Immediate Roadmap:

• Implement semantic similarity search for finding related memories


• Add emotional trend analysis across time periods


• Create memory timelines with contextual insights

Future Enhancements:

• Voice emotion detection for enhanced emotional context


• Collaborative memory sharing with privacy controls


• Integration with calendar and productivity apps


• Advanced pattern recognition for personal growth insights

Final Note

Whispers demonstrates how Algolia MCP Server can transform simple text search into contextual memory recall. By combining structured data indexing, intelligent query decomposition, and semantic relevance scoring, it creates a second brain that understands not just what you said, but the context, emotion, and patterns in your thoughts over time.

The project showcases how MCP technology enables applications that feel like they understand you—not just search your data, but help you rediscover and reflect on your own thoughts and growth journey.